Forth-eV Wiki

User Tools

Site Tools

You are here: Forth Wiki » Forth-ev Wiki » pfw » NOF file system, for small Flash chips
Trace:
en:pfw:file_systems

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
en:pfw:file_systems [2023-09-04 18:14] – gelöscht - Externe Bearbeitung (Unbekanntes Datum) 127.0.0.1en:pfw:file_systems [2024-05-09 16:46] (current) – [NOF structure] willem
Line 1: Line 1:
 +===== NOF file system, for small Flash chips =====
 +
 +----
 +
 +==== NOF idea ====
 +
 +Read and store data from/to source files on some sort of background medium. 
 +For these purposes a file system could be used. 
 +There are lots of variants present: ''%%FAT16, FAT32, NTFS, Ext2, Etc.%%''\\
 +This design is about a compact file system for small controllers and Flash memory chips. 
 +It uses [[en:pfw:crc16|16-bit CRC-codes]] instead of file names and directory names.
 +NOF was developed on top of noForth and stands for **nO**Forth **F**ile system.
 +It can ofcourse be used on any system with limited resources.
 +----
 +
 +==== NOF structure ====
 +
 +  * The reading is done on a per-byte basis
 +  * Write goes in 256-byte sectors
 +  * Erasure is done with sectors of 4096 bytes, or ... the entire chip at once
 +  * The first 4096 bytes is the ''%%FET%%'' (**F**ile **E**ntry **T**able)
 +  * The ''%%FET%%'' sector contains the ''%%FID%%'' (**F**ile **ID**) & ''%%DID%%'' (**D**irectory **ID**)
 +  * A file has a size of 4096 bytes or a multiple of that
 +  * Each file-ID is 8-bytes, a directory-ID is 16-bytes
 +  * The first 32 bytes of a file contains the file name, ''FIT'' (**FI**le **T**ype) and an auxillary byte
 +
 +A 2-Mbyte Flash has 512 pages of 4096 bytes. 
 +When the first sector is used for the ''%%FET%%'', 511 pages are free for files.
 +The ''%%FID%%'' gets 3072 bytes, the ''%%DID%%'' the remaining 1024 bytes.
 +This means there is space for 384 files and 64 directories. 
 +When you are in need for more files and directories, just take the first two or more (erase)-sectors
 +of the Flash chip for administration.
 +
 +**File ID record**
 +<code>
 +        2 Bytes        2 Bytes      2 Bytes        2 Bytes (8 bytes)
 +    ----------------------------------------------------------
 +    | sector number | dir hash | filename hash | file length |  ( max. 384 files )
 +</code>
 +**Directory ID record**
 +<code>
 +       2 Bytes       2 Bytes        12 bytes  (16 bytes)
 +    -------------------------------------------------
 +    | Dir-hash | Working-Dir-hash | Dir-name string |  ( max. 64 directories )
 +</code>
 +**File header & first file record**
 +<code>
 +      Byte Byte      30 bytes           4064 or more bytes
 +    |-------------------------------| --------------------- |
 +    | FIT | Aux | Filename 30 bytes | File data             |
 +</code>
 +
 +==== NOF basic implementation parts ====
 +
 +  - An SPI-driver & setup for the Flash chip
 +  - Operating commands for the Flash chip (Write, Read, Erase, etc.)
 +  - Some tools (''%%FDUMP%%'' dump flash memory)
 +  - Multiple file buffers
 +  - Directory stack
 +  - ''%%MOUNT%%'' function
 +  - Convert a string to 16-bits CRC
 +  - Choose & show directory
 +  - Find file & include file
 +  - Write, Read & Erase given sectors
 +  - Build ''%%FID%%'' entry & file name block
 +
 +Later on some more additions, like:
 +
 +  - Store ASCII files
 +  - File viewer
 +  - Remove files, wipe complete disk
 +  - Make directory and a directory ''%%TREE%%'' function
 +
 +{{pfw:nof_at_work-1.jpg}}
 +==== Pseudo code for the NOF SPI layer ====
 +
 +More detailed info on SPI all numbers in hexadecimal  ([[en:pfw:spi]])
 +
 +<code forth>
 +Function: FSPI-OUT   ( b -- )  \ Output a byte using the SPI-bus
 +Function: FSPI-IN    ( -- b )  \ Read a byte from the SPI-bus
 +Function: FSPI-SETUP ( +f -- ) \ Initialise SPI-bus to Flash with frequency +f
 +Function: {FL        ( b -- )  \ Enable Flash access & send first byte 
 +Function: FL}        ( -- )    \ Close Flash access
 +
 +Function: {FREAD     ( a c -- b )
 +    {fl   split 32bit into two 16bit hi part on top   fspi-out
 +    split 16bit into two 8bit part hi part on top   fspi-out
 +    fspi-out   fspi-in
 +   
 +Function: FREADY?    ( -- f )
 +   5 {fl   spi-in   fl}  1 and 0=
 +
 +Function: WRITE-ON   ( -- )         6 {fl  fl}
 +Function: BUSY       ( -- )         begin  fready? until
 +Function: CHIP-ERASE ( -- )         write-on  60 {fl  fl}  busy
 +Function: POWER-UP   ( -- )         66 {fl  99 fspi-out  fl}
 +Function: FC@        ( a -- b ) 3   {fread  fl}
 +Function: FC@+       ( a -- a+1 b ) dup 1 + swap fc@
 +Function: F@         ( a -- x )     3 {fread  fspi-in  fl} make 16-bit
 +Function: F@+        ( a -- a+2 x ) dup 2 + swap F@ 
 +Function: FTYPE      ( a u -- )     0 ?DO  fc@+ emit  LOOP  drop
 +
 +Function: ID.        ( -- )         0 90 {fread  fspi-in  fl}  .  .
 +   Save number base & set it to decimal
 +   
 +Function: FDUMP      ( a u -- )
 +    <freq> fspi-setup  power-up
 +    0 ?DO   new line and print address followed by ':'
 +            show 10 bytes from flash memory in hexadecimal
 +            show 10 bytes as ASCII or a space when not a valid character
 +            Add 10 to address
 +            Stop when a key was pressed
 +    LOOP drop address
 +
 +</code>
 +
 +==== Pseudo code for the NOF structure layer ====
 +{{{{pfw:nof_-_fet.jpg}}}}
 +<code>
 +
 +Function: RWDATA      ( A data structure of 7 cells for the NOF structure layer )
 +Function: !DATA  ( x +n -- )  Store x op position +n in RWDATA
 +Function: @DATA  ( +n -- x )  Read x from position +n in RWDATA
 +
 +0000 value #SECTOR    ( Flash sector counter, initialised by MOUNT )
 +0000 value BUF        ( Buffer in use  0/1 )  
 +0000 value #N         ( Byte index in sector buffer )  
 + 200000 100 / constant #FLASH ( Minimal flash size, max: 1000000 )
 +0000 constant #FID    ( Begin address of file ID block )
 +0C00 constant #DID    ( Begin address of directory ID block )
 +1000 constant #FILES  ( Begin of file area )
 +
 +Function: INIT-RWDATA  ( -- )  
 +    Write flag is zero, collected data is zero,  
 +    store zero in #n and select buffer-0  
 +
 +Function: 'NAME  and reserve 20 bytes for string to convert
 +create DID and reserve 0C bytes RAM after it  
 +value DPT   ( Directory nesting pointer )
 +value 'DID  ( Free directory space pointer )
 +value 'FID  ( Free file index pointer )
 +value HASH> ( Hash working space )
 +
 +Function: .DFREE     ( -- )
 +    Save number base & set it to decimal
 +    Subtract #SECTOR from #FLASH and divide by 4
 +    to convert from sectors to kBytes, print it & restore base
 +
 +Function: ROOT   ( -- )  0 did h!   0 to dpt
 +Function: DID>   ( -- )  dpt IF  did dup 2 +  swap dpt move  DPT = DPT + 2  THEN
 +
 +Function: $>UPC  ( a1 u -- a2 u )
 +    Store string a1 u1 at 'name, save length
 +    Convert string in 'name to uppercase
 +    leave address of converted string & length
 +    
 +Function: IHASH  ( h -- )          Store h in HASH>
 +Function: >HASH  ( c -- )          Read HASH> xor char c do result times 2 and store in HASH>
 +Function: HASH   ( a u h0 -- h1 )  Generate file hash h1 from given string a u & hash seed h0
 +Function: DHASH  ( 'did h0 -- h1 ) Generate dir hash h1 from string stored in 'did+2 & hash seed h0
 +
 +value FADDR
 +Function: FKEY   ( -- c )
 +    faddr fc@  increase faddr  ?dup ?exit  -2 to source-id  0D ;
 +
 +Function: MOUNT  ( -- )
 +     <freq> fspi-setup  initialise R/W data structure
 +     Token of FKEY to alternative KEY-vector
 +     Initialise 'FID by scanning the FID memory until FFFF is found
 +     Initialise 'DID by scanning the DID memory until FFFF is found
 +     Initialise first free file sector from reading the last found file entry
 +     store it in #SECTOR  finally set directory to the root
 +</code>
 +
 +==== Pseudo code for the NOF directory mechanism ====
 +{{pfw:nof-dir.jpg}}
 +<code>
 +Function: CHECK-DHASH  ( h -- f )
 +    Save current DIR-entry address
 +    #did 'did over - bounds DO  ( Loop true directory space )
 +        dup i f@ = IF           ( Is there a hash match? )
 +            replace dir-entry address
 +            Leave true and leave loop immediately
 +        THEN
 +    10 +LOOP                    ( To next DIR-entry )
 +    drop false                  ( No directory found )
 +    
 +Function: .DIR  ( did -- )
 +    When did is zero its the root show that ans ready
 +    check-dhash IF
 +        Read dir-entry address and print it's name
 +    THEN
 +
 +Function: .PATH  ( -- )
 +    Get DID  read DPT pointer and print
 +    the current directory nesting in correct order
 +
 +Function: >DHASH  ( a u -- h1 h0 )
 +    Limit string to 0E characters ( 00 to 0D )
 +    Generate root hash value h0 and current DIR hash value h1, leave both
 +
 +Function: CHOOSE-DIR  ( a u -- )
 +    >dhash  was it a backslash then select ROOT
 +    Is it a dot then go back one directory nesting when possible
 +    check-dhash and abort when it's an invalid directory
 +    Otherwise extend directory nesting and save current DIR record address
 +
 +Function: CD  ( "dir" -- )
 +    Set a new directory nesting by unraveling the given string by parsing it with 
 +    backslashes. When there is a backslash in front start in the root.
 +    When the string is done, show the new directory path using .PATH
 +
 +Function: .FIT  ( +n -- )
 +    Print file type +n as an ASCII string, +n comes from the 20 bytes file header
 +
 +Function: DIR  ( -- )
 +    1) Print current dir path with .PATH
 +    2) Read current directory entry from DID and print all
 +       sub)directories in it by calculating the nested hash with DHASH
 +       Print the directory names that give a valid hash code
 +    3) Now print the files names, by checking if it's dir-hash code is the
 +       same as the one stored in DID if so print the file type and the name 
 +       from the 20 bytes file header
 +    4) Finally show the free space on the Flash disk using .DFREE
 +</code>
 +
 +==== Pseudo code for the NOF file search function ====
 +{{pfw:nof_-_fid.jpg}}
 +<code>
 +Function: CHECK-FHASH  ( h -- f )
 +    #fid  'fid bounds ?DO          ( Setup search range )
 +        Read dir hash value from second cell in FID entry 
 +        is it equal to the contents of DID
 +            IF  Read the the file hash from the third cell
 +                when equal, save this FID record
 +                leave true and were ready
 +            THEN
 +        THEN
 +    8 +LOOP                         ( To next FID entry )
 +    file hash not found, leave false
 +        
 +Function: >FHASH  ( a u -- h )
 +    Limit string to 1D chars, do:  $>upc DID h@ dhash
 +    
 +Function: 'SEEK?  ( a u -- sector true | false )
 +    >fhash  check-fhash IF
 +        Read the files start sector, leave true & ready
 +    THEN  print "File not found" & leave false
 +    
 +Function: LOADFILE  ( a -- )
 +    source-id greater then zero, nest file address pointer
 +    nest source-id  and store a to faddr
 +    set source-id to 1
 +    call interpreter, when done
 +    unnest source-id
 +    when source-id is greater then zero unnest faddr & save
 +
 +Function: SEEK  ( "name" -- sector )
 +    read blank delimited string "name"
 +    'seek?  issue error message when result was zero
 +    
 +Function: INCLUDE  ( "name" -- )  immediate word
 +    abort when used in a definition
 +    read blank delimited string "name"
 +    check if it's a forth file ans issue an error message when not
 +    Calculate the files address & call: loadfile
 +    
 +Function: FILE  ( "name" -- a )  immediate word
 +    seek  convert sector to file address 
 +    and compile it as a literal
 +    
 +</code>
 +
 +==== Pseudocode for creating a file header ====
 +
 +{{pfw:nof-file-header.jpg}}
 +<code>
 +Create: 'SECTOR  200 allot  ( -- a )    ( Reserve RAM for two sectors )
 +Define: ADDR-SECTOR  ( sa -- )
 +    split sa in low- & high-byte  fspi-out  fspi-out  0 fspi-out
 +    
 +Define: BUFFER  ( +b -- a )
 +    1 and  100 *  'sector +
 +    
 +Define: READ-SECTOR  ( sa +b -- )
 +    >r  3 {fl  addr-sector  r> buffer
 +    100 0 do  fspi-in  over c!  1 +  loop  fl}  drop  
 +
 +Define: WRITE-SECTOR  ( sa +b -- )
 +    >r  write-on  2 {fl  addr-sector  r> buffer
 +    100 0 do  count fspi-out  loop  drop  fl}  busy 
 +
 +Define: ERASE-SECTOR  ( sa -- )
 +    write-on  20 {fl  addr-sector  fl}  busy
 +
 +Define: B,  ( b -- )
 +    #n 100 = IF  set buffer full flag,  select next buffer and clear #n  THEN  
 +    store b in current buffer & increase pointer #n
 +
 +Define: WR-BUFFER  ( -- )
 +    Write buffer-0 to #sector and increase #sector
 +
 +Define: <WRITE>  ( -- )
 +    When buffer overflow has occured do: wr-buffer
 +    and clear buffer full flag, copy buffer overflow
 +    from buffer-1 to buffer-0, slect buffer-0 again
 +
 +Define: !FLENGTH  ( -- )    ( Patch current rounded file length in last FID record )
 +    Read last used sector, round it to next 10th sector
 +    Read current part of FID sector to buffer-1
 +    Patch calculated file length into it
 +    Write buffer-1 back to current FID sector
 +
 +Define: FILE-NAME  ( #fit a u -- )
 +    init-rwdata  limit string to 1D ans make uppercase
 +    Calculate & check file hash value, abort when the hash value is not unique
 +    Calculate, save & read current FID-sector in buffer-0
 +    Save address of new FID record too
 +    Store DID in record, after that the file-hash
 +    Now write FID sector back & add 8 to 'FID
 +    Now store file type on address 0 in buffer-0
 +    after that the (limited) file name string
 +    Set #n to 20 and ready
 +
 +Define: ADD-FID  ( #fit "name" -- )
 +    Read next word from input stream and build a new file entry with it
 +</code>
 +
 +<code></code>
 +
 +==== NOF files ====
 +
 +The added NOF example files are for noForth R on the GD32VF103, especially the [[https://www.seeedstudio.com/SeeedStudio-GD32-RISC-V-Dev-Board-p-4302.html|seeed board]] which has a W25Q64 (8 Mbyte) Flash chip added on the board!
 +
 +| File name | Purpose | in Dropbox (external link) |
 +| [[https://www.dropbox.com/s/gqpzi24n764cmjh/SPI%20Flash-03.f?dl=1|SPI Flash-03.f]] | NOF SPI layer | [[https://www.dropbox.com/s/gqpzi24n764cmjh/SPI%20Flash-03.f?dl=0|SPI Flash-03.f]] |
 +| [[https://www.dropbox.com/s/gqpzi24n764cmjh/SPI%20Flash-03.f?dl=1|NOF-3i.f]] | NOF base file | [[https://www.dropbox.com/s/gqpzi24n764cmjh/SPI%20Flash-03.f?dl=0|NOF-3i.f]] |
 +| [[https://www.dropbox.com/s/b6xagirqy8b6lj9/NOF-write-1.f?dl=1|NOF-write-1.f]] | NOF write | [[https://www.dropbox.com/s/b6xagirqy8b6lj9/NOF-write-1.f?dl=0|NOF-write-1.f]] |
 +| [[https://www.dropbox.com/s/bv2sqamfeuvh7bd/NOF-additions-1.f?dl=1|NOF-additions-1.f]] | NOF additions | [[https://www.dropbox.com/s/bv2sqamfeuvh7bd/NOF-additions-1.f?dl=0|NOF-additions-1.f]] |
 +
 +[[en:pfw:welcome|Back to PFW page]]