Research & Development

MemoryLayout | Apple Developer Documentation

Useful documentation for MemoryLayout

Started working on memory mapping with a really long mistake by working with hnsw as a HNSWCorpus in the ContextFreeEncoder leading to a circular dependency.

Added Codable conformances with encoding and decoding for HNSWCorpus (unused) and DeterministicSampleVectorIndex for requirement in ContextFreeEncoder with file I/O. Should correctly set the foundation for reading and writing the HNSW index to and from a memory map.

Added extra PreComputedEmbeddings enumeration for hnswindex for the corresponding memory map file. Added corresponding changes to initializer.

Minor improvements and bug fixes:

Fixed computed field count in HNSWCorpus
Moved zeroes to computed field in HNSWCorpus
Matched initializers of HNSWCorpus to DictionaryCorpus
Added typicalNeighborhoodSize field to DeterministicSampleVectorIndex with default 20

Tests:

Added memory usage logs for HNSWCorpus, interestingly the corpus is only 8 bytes for both the basic and small test case
Added basic query tests for HNSWCorpus in HNSWTests, it successfully finds vectors but it returns it in a raw format
Added test cases with typicalNeighborHoodSize variations, but oddly crashes after 0
Added pipeline tests similar to AllMiniLM_pipelineTest.swift, unable to find dependency for TestUtils

Questions

I am currently setting the width (assuming this is dimension) to be 50 in ContextFreeEncoder for the HNSW index. How should I go about verifying what the actual dimension is?

You don’t. Everything in ContextFreeEncoder doesn’t need to be touched. We are focused on using glove embeddings for now.

I have the foundation ready for the memory map and I understand I am supposed to store cache the index, but where do I do this (ie. what file)? Additionally, on what variable/data structure should I do this on?

Create an extension on HNSWCorpus which allows us to read and write to a file. We will likely need to write it into a Data object with .alwaysMapped.