Best/biggest/easiest way to store data in RAM or EEPROM

[tumbleweed]

I think I was wrong about the density. If you're looking for higher density devices then SPI NAND flash is probably the better choice, but it's much different. It has a different command set, and you have to deal with potentially bad blocks.

You won't get a serial static RAM in that density either. Cost-wise your best bet is an SD card.
 

[Amateurtje]

The SD card script I have is not really fast. And I think it is relatively old.

Another idea is to leave 1 SD card on the PCB so The data is at least save......

I now realise: I was writing to a text file... MAybe better to write directly to the sd.. Is that possible?

Does anybody have a good example...??

Is this a better way to go? 2 SD cards where 1 stays in the device?

[tumbleweed]

MAybe better to write directly to the sd.. Is that possible?

Yes, but if you want to be able to read the data with something else then you need to use a filesystem (FAT, etc).

Formatting the data as a text file instead of a binary data file will slow things down and increase the amount of storage needed, but it has the advantage that you can read it directly without needing some app to interpret the data for you.

I don't know why you'd need two... unless you pull the card while it's being written to, so don't do that!

[Amateurtje]

I will explain my thoughts.

My current setup is not reliable in writing and is slow and writes it to a txt file on the SD card. I had it already a couple of times that the data was lost and the txt file was damaged... (eventhough I waited nicely when pulling it out..

So the idea:

SD-Card1 is permantly in the device (as EEPROM, While that seems to be difficult)
Store the data on SDcard1 directly. That is faster, so during logging it stores on Sd-card1. That data should be more save.

Then, when the logging is done and I want to read the data, I press a transfer button, Letting the device read the data on SD-Card-1 and placing this data in a text file on SD-Card2. When this transfer is done, SDcard2 (with txt file) can be taken out en read in a pc..

The data in the device needs to/can be deleted when the user has used the data.

So, in the end, the same setup as I wanted to do with the EEPROM, but using instead of the EEPROM a second SD-CARD...

Something like this maybe a good idea?

[tumbleweed]

I suppose so, but if your current setup is not reliable then you have something wrong in your code/SD library.
If you have a "stop logging" button and you press that, wait for some visual feedback (like an LED) that says it's safe to remove the card.

I'd get that working first since it sounds like you already have all that hardware.

You're not doing this in an interrupt routine by chance, are you?

[Amateurtje]

No, it is not writing in an interrupt. But it is also slow while writing which does influence the read-speed and GPS analisys.. So speading up the writing of the data is also on the wishlist.

In the interrupt, it is now reading the incoming GPS data and it is receiving serial data that nees to be logged. In the future, a lot of other  sensors will be attached.

I use the SD card scripts which I found here:

Code Select Expand

;-----------SUB ROUTINES-----------------------------------------------

Insertion:
    Print At 1,1,"inserer la carte"
   
    Repeat
        Response = SD_Init_FS               '(version SPI Soft)     
    Until Response = 0
    Print At 1,1,"SPI soft OK     "
    pause
 
DelayMS 1000 
Return

Taille:     
    Print At 1,1,"verif Taille disque"
    Disk_Size = SD_Disk_Size
    Print At 1,1, "Taille:",Dec Disk_Size," KB    "
    pause
DelayMS 1000
Return
 
 Consulte:
    Print At 1,1,"Ouverture       "
    SD_File_Name = "CONFIG"                                'File Name, upper case only!
    SD_File_Ext = "TXT"                                     'File Ext, upper case only! 
    Response = SD_Open_File
    If Response = 1 Then
        Print At 1,1,"Fichier absent  "
        Stop
    EndIf
    pause   
DelayMS 1000   
    Print At 1,1,"Fichier ouvert  "
    pause
    While SD_EOF < 1
        Response = SD_Read_Byte_From_File
        If SD_EOF < 1 Then Print At 1,1,Response
    Wend
    pause     
 DelayMS 1000
Return
 
 Ajout:
    SD_File_Name = "DATA"
    SD_File_Ext = "TXT"
    Response = SD_Append_File
    If Response > 0 Then
        Print At 1,1, "pas de fichier !"
        Stop
    EndIf
   
    Print At 1,1, "ajout en cours  "
    pause
    SD_IO_String = "udootje" + 13 + 10
    Response = SD_Write_String_To_File                      'Write SD_IO_String to file
    SD_Close_File
    Print At 1,1,"Fermerture......"
    pause     
DelayMS 1000
Return

Ouvrir_Nouveau:
    SD_File_Name = "TEST"
    SD_File_Ext = "TXT"   
    Response = SD_Check_For_File                            'Check if file already exists
    If Response = 0 Then
        Print At 1,1,  "existe deja !   "
        pause
        SD_Delete_File
        Print At 1,1,"effacement...   "
        Stop
    Else         
        Print At 1,1, "creation..      "
        SD_New_File                                         'Open new file
    EndIf
    pause
DelayMS 1000
    Print At 1,1,"ecriture....    "
    SD_IO_String = "bourrage **********************************************************"
    SD_Write_String_To_File                                 'Write SD_IO_String to file
    SD_Write_Byte_To_File 13                                                   
    SD_Write_Byte_To_File 10
    SD_Close_File
    Print At 1,1,"Fermerture......"                          'Close file
    pause
DelayMS 1000
Return

Schrijven1:
    Response = SD_Write_String_To_File                      'Write SD_IO_String to file
   SD_IO_String =SD_IO_String1
    Response = SD_Write_String_To_File   
    SD_IO_String =SD_IO_String2   
    Response = SD_Write_String_To_File   
     SD_IO_String =SD_IO_String3     
    Response = SD_Write_String_To_File   
     SD_IO_String =SD_IO_String4   
    Response = SD_Write_String_To_File 
         SD_IO_String =SD_IO_String5     
    Response = SD_Write_String_To_File     
Return



Adn the Schrijven1 script I call in the main loop. There it processes the GPS info, Processes the incoming data (in the future processes the sensors) and then writes it to SD...

but, I also just can not take the risk the data is lost, so storing it with and transferring it later is really not a bad idea.... I will design for the eeporm (or SD-card1 ) just a new pcb and order it... But first will make a kind of addition to my pcb as i did now also for the EEPROM...


So I rather put my time in first logging it  (and that can be with this pcb) to the SD-card but directly, so not in a textfile...

Does that work also with adressing??

[trastikata]

Hello Amateurtje,

You really should take the time to read the datasheet and understand what is happening in my code!

The code I posted is to give you an idea how to deal with the FLASH chips!

I only glanced at MT29F1G01 and I see the command set is almost the same name-wise, but has different values, however you used the command values from my code, not even checking ...

Reset is 0xFF instead of 0x66 + 0x99

Only Block erase is available and it is 0xD8 instead of 0x20

No one will read the datasheet and write the code for you, MT29F1G01 is not different in the way you manipulate it from other FLASH chips, thus take some time and study the datasheet.

[Amateurtje]

Hi Trastikata,

Oke, so you really think it is possible with that chip... That gives hope. I will for sure dive back into it...

No one will read the datasheet and write the code for you, MT29F1G01 is not different in the way you manipulate it from other FLASH chips, thus take some time and study the datasheet. 

Yes, I know,        ..  I am so short for time, And I sometimes am thinking about letting other people do it....mmm.. IS there nobody on this forum who wants to earn a couple of euro's? I am not a rich guy but sometimes I have even less time to complete things (which I promised to people..  )
 
With familiar stuff it is easier to read datasheets while you know what you are looking for... With this, I first really need to understand how it all works.. I got the big lines of the code but probably not the details...


What you said about trying to read the data which is already inside, seems a good way to start...

[trastikata]

We are all short of time I think, it's a matter of priorities 

I don't know what are you building but 3840 bytes per second for data logging seems quite excessive.

[Amateurtje]

We are all short of time I think, it's a matter of priorities 

Absolutely true...But I can always hope to find somebody    

I don't know what are you building but 3840 bytes per second for data logging seems quite excessive.

I took a lot of reserve places but indeed, i want to log a lot of data. butbeteer to have some reserve than to come short when you are finished building it.....
And looking at the fact that it is not easy to find a big working eeprom, it is good to hope for a big one... Going from the assumption that it is 1 GByte to find out it is 1 Gbit, it is already a factor 8 that i need to cover.. so all in all, yes, it is a lot of data but if I start to minimise on all sides the data and the wished specs, it will still be a big amount of data ..   

[tumbleweed]

MT29F1G01 is not different in the way you manipulate it from other FLASH chips

Being a NAND flash it is different. You have to deal with ECC and bad blocks.

From the datasheet...

This NAND Flash device is specified to have the minimum number of valid blocks (NVB)
of the total available blocks per die shown in the table below. This means the devices
may have blocks that are invalid when shipped from the factory. An invalid block is one
that contains at least one page that has more bad bits than can be corrected by the min-
imum required ECC. Additional bad blocks may develop with use. However, the total
number of available blocks will not fall below NVB during the endurance life of the prod-
uct.

Although NAND Flash memory devices may contain bad blocks, they can be used relia-
bly in systems that provide bad-block management and error-correction algorithms.
This ensures data integrity.

[top204]

Because of the price and size (in memory) of the micro SD memory cards, they are a very good method of NVM, and can easily be fitted to a PCB.

When writing to an SD card, always use a double buffer method, so as the data is being written to it, more data is being stored in a RAM buffer, then once the RAM buffer is full, they are swapped and the process happens again. With this method, very high speeds can be achieved when writing, and if faster speeds are required, use a triple buffer mechanism. All it takes is 512 bytes per RAM buffer, so a double buffer will require two arrays of 512 bytes each.

If it is only the unit that is accessing the NVM, there is no need for a FAT system, and it is then even faster to write to the sectors themselves (512 bytes at a time with the buffer mechanism), and a serial interface to the unit can transfer the data from the sectors into the computer. Again using the double buffer mechansim for high speed reading with a slower interface mechanism such as serial.

When using an SD card on the PCB, always have a method of resetting it with a P-Channel MOSFET connected to its VDD line, so if it fails to initialise or write, toggle the power to it to reset it.

I've used SD cards as NVM in a few projects and they work well, and I have even accomplished 44KHz, 16-bit stereo audio recording/playback on them with an 18F device operating at 64MHz and a simple triple buffer mechanism, and two 8-bit PWM channels combined to give a 16-bit resolution with a couple of resistors for each channel (with a resistance ratio of approx 256:1), and over-sampling the 12-bit ADC to 16-bit. This method is something I am hoping to make an article about on my google compiler site, because the compiler produces extremely efficient code for the conversions, and the high speed PWM produces an excellent 16-bit DAC for very good quality sound for very little expense.

[tumbleweed]

That would be a great article, Les.

Looking forward to reading it...

[trastikata]

T29F1G01
Being a NAND flash it is different. You have to deal with ECC and bad blocks.

Not in this case tumbleweed, the ECC correction is automatic and enabled by default.

During a PROGRAM operation, the device calculates an expected ECC code on the ECC-
protected bytes in the cache register, before the page is written to the NAND Flash array.
The ECC code is stored in the spare area of the page.
During a READ operation, the page data is read from the array to the cache register,
where the ECC code is calculated and compared with the expected ECC code value read
from the array. If a 1–8-bit error is detected, the error is corrected in the cache register.
Only corrected data is output on the I/O bus. The ECC status register bit indicates
whether or not the error correction is successful.

And it is doubtful if  one will get over the endurance life if used as normal FLASH which has the same amount of write cycles. Sure bad blocks management can be required but probably not in the first years of normal use.

[tumbleweed]

The correction can be automatic, but dealing with bad blocks isn't.

The factory identifies invalid blocks before shipping..

The parts are likely to come direct from the factory with bad blocks... that's acceptable.
Check the Nvb spec.

NAND flash is different and shouldn't be treated like NOR flash.

[trastikata]

The parts are likely to come direct from the factory with bad blocks... that's acceptable.
Check the Nvb spec.

I'll, thanks.

[Amateurtje]

Hi All,

You all helped me a lot (thanks!!) but also gave me a difficult decision  . I tihnk i go the route of the SD card (actually a double sd card). One "Fixed" for writing data directly and one for creating on a button press a textfile that can be used in the pc. (and uploaded to a website for analysis).

Ofcourse I will try to follow the double buffer method as described by Les, as soon as I understand exaclty how to make that .. That article would be increadibly nice but we all know how busy Les is ...

Anybody already written directly to an SD card? Any starting points?

then once the RAM buffer is full, they are swapped and the process happens again.

How do you mean this swap? While I understand that the SD card works in blocks of 512 byte, I understand to write in chucks of 512 byte. But it is still a writing actin that takes time or do you see a method to do it parrallel? I was reading somewhere about using a second core on the processor but that does not seem logical using a pic... (specially not a 18F46k22 )

but do you just mean, put data in the ram, When you have a chunk of 512 byte, write it to the SD card...

While I have a lot of data, I can easily make chunks of 512 byte...

Or am i missing the idea completely..

[top204]

The data is written from the sensor/s to a RAM buffer, then when the RAM buffer is full, its contents are written to the NVM and the RAM buffer that is written too is flipped to another one, so while one is being filled with data, the other is being written to the NVM, then the RAM buffers are flipped when ready to write their contents.

A code snippet from an Interrupt handler using the CCP2 peripheral as the time is listed below. It is from an 8-bit audio record/playback project I did a few years ago:

Code Select Expand

'
' Compare Special Event interrupt handler
' Used for recording audio
'
    If PIR2bits_CCP2IF = 1 Then                                 ' Was it a Compare Special Event on CCP2 that triggered the interrupt?
        If Global_tRecording = True Then                        ' Yes. So are we recording?
            If Global_tRAM_BufferNum = cUse_RAM_Buffer0 Then    ' Yes. So are we writing to Global_RAM_Buffer0?
                Audio_wFSR0 = AddressOf(Global_RAM_Buffer0)     ' Yes. So load the address of Global_RAM_Buffer0 into FSR0L/H
            Else                                                ' Otherwise. We're writing to Global_RAM_Buffer1 So....
                Audio_wFSR0 = AddressOf(Global_RAM_Buffer1)     ' Load the address of Global_RAM_Buffer1 into FSR0L/H
            EndIf
            Audio_wFSR0 = Audio_wFSR0 + Global_wPageIndex       ' Add the location within the RAM buffers to FSR0L/H
            Audio_ADC_Get(INDF0)                                ' Get an 8-Bit ADC reading into the current RAM buffer

            Inc Global_wPageIndex                               ' Increment the RAM address
            If Global_wPageIndex >= cFlash_PageSize Then        ' Have we reached the end of a RAM Buffer?
                Global_wPageIndex = 0                           ' Yes. So reset the page index
                Toggle Global_tRAM_BufferNum                    ' Move to the next RAM Buffer (0 is Global_RAM_Buffer0, 1 is Global_RAM_Buffer1)
                Global_tTimeToWritePage = True                  ' Indicate to the main program that the current RAM Buffer needs writing to the flash
            EndIf
        EndIf
        PIR2bits_CCP2IF = 0                                     ' Clear the interrupt flag
    EndIf

I know it looks complex, but the lower case tags on the variable names will show what type they are, and it can be seen that for a double RAM buffer, a single bit "Global_tRAM_BufferNum" is set to 0 for RAM buffer 0, or 1 for RAM buffer 1, and once the RAM buffer is filled and ready to write to the NVM, the bit is toggled and a flag is set to tell the main program loop to write the RAM buffer to NVM.

The RAM buffers are declared as:

Code Select Expand

'
' Create RAM buffers for the flash memory writing
'
    Dim Global_RAM_Buffer0[cFlash_PageSize] As Byte Heap       ' Create a RAM array at the end of the above variables
    Dim Global_RAM_Buffer1[cFlash_PageSize] As Byte Heap       ' Create a RAM array at the end of the above variables

Where cFlash_PageSize is the size of the NVM's page, so it can be written all at once.

Hopefully, the code will give you an idea how straightforward double buffering is. For triple, or more, buffering, a byte variable is incremented to indicate what RAM buffer is available to write too/from, instead of a bit variable.

[Amateurtje]

Hi Les,

Yes, I get it.. seems relatively straight forward...I think I need to do that with a bigger buffer and use buffers of multiple blocks of 512 byte..

The part of writing to the NVM you did not put unfortunatly.. I will go over the forum to look if I can find it. Or can I do it with the basic script of Trastikata? I have no clue how and which adresses ate but I will see if I can find something..

[Amateurtje]

After some time I picked up this again and I thought I got it to work but it seems to go very wrong.

I have made the below test script, and I want to write directly to the sectors on a SD card. The Subs that are used are the unchanged subs for writing to an SD card, therefore I do not post them here. On request, i can do that ofcourse.

"Print At 1,9,Dec3 SD_Buffer0_Intern[1]"
On the display, during the writing, it displays on 1,9 only 233. The last written value. It looks that he makes all secotors the same??
Due to the variable "a", it should incrrease the values by 1, and I see that happening during the writing phase.

Code Select Expand

DelayMS 1000 
Cls

Main1:
Cls
Print At 1,1, "Start"
DelayMS 1000
Cls
Print At 1,1, "Writing." 

Dim z As Dword
a= 0
For z = 256  To 1000
    For i=0 To 255
        SD_Buffer0_Intern[i] =i +a
        SD_Buffer1_Intern[i] =i
    Next i
    SD_Address_Intern  = z
    GoSub SD_Write_Sector_Intern
    a=a+1
    Print At 1,9,Dec3 SD_Buffer0_Intern[1]
    Print At 2,1, Dec6 z
Next z

DelayMS 1000
Cls
Print At 1,1, "Reading." 
  DelayMS 1000
For z = 256  To 1000'' step -1
    SD_Address_Intern  = z
    Print At 2,1, Dec6 z
    DelayMS 50
    GoSub SD_Read_Sector_Intern
    DelayMS 100
    Print At 4,1, Dec6 z
    DelayMS 50
    Print At 1,9,Dec3 SD_Buffer0_Intern[1]
    DelayMS 100
'    For i=0 To 255'' step -1
     
'      print at 3,1, Dec3  SD_Buffer0_Intern[i]  , " ", Dec3  SD_Buffer1_Intern[i]
'      delayms 50 
'    Next i
Next z
Cls
Print At 1,1, "end." 
DelayMS 1000
GoTo Main1


End
I will try to get more info, but does anybody have an idea? Can anybody help me? Is it possible to write like this, diffrenet info to the sectors?