This actually works. I'm using it in an audio recording/looping app where I need to read the data from arbitrary locations while it's being recorded. I haven't tested it with encoded audio, but it works great for LPCM.
For writing:
AudioFileWritePackets takes an offset argument so there is no "seeking".
AudioFileID audioFile;
AudioFileCreateWithURL(url, kAudioFileCAFType, &asbd, kAudioFileFlags_EraseFile, &audioFile);
AudioFileWritePackets(audioFile, 0, samplesToWrite * sampleSize, NULL, offset, &samplesToWrite, inBuffer);
For reading you use the same AudioFileID to create your ExtAudioFileRef:
ExtAudioFileRef extAudioFile;
ExtAudioFileWrapAudioFileID(audioFile, 0, &extAudioFile);
ExtAudioFileSeek(extAudioFile,readStartSample);
ExtAudioFileRead(extAudioFile, ioNumberFrames, bufferListOut);
It don't know if the CAF file type is necessary or not, but it's a good starting point.