Bufferless averaging

[TimB]

Hi all

I'm using a pic that is a bit anemic when it comes to RAM. I'm running out quick but need to perhaps implement a buffer to get an moving average of a couple of values

As an overall picture of a what my situation is

I have an optical sensor that is measuring the presence of a fluid in a clear pipe. I'm using a comparator in the pic to get very high speed samples and free up the ADC converter for other jobs. So far so good

But I need to adjust the threshold that comparator switches at. This I can do using a DAC in the pic So far so good.

The issue is that over time the tube etc will get dirty, the led in the optical sensor will decrease in output. This in the long run will lead to the threshold needing adjusting. I have the output of the sensor also ties to an ADC port that I can sample at various times to get a reading. The issue is that I do not know what the reading represents. Sure with enough ram I could log a number of the lowest values (Y) and a number of the highest (X) then Z = X-Y and Z / 2 and A = Y+Z

Now this is the issue, I want the average Highs and average Lows. Not just the highest and lowest I have seen. As if the unit is running for a very long time those values will become out of date. I could have a big ring buffer and try and do some analyse on that.

BUT as I started with I'm low on ram, I have about 30 bytes left. My ADC readings are words so not much of a buffer

What I want is a system to get a running average of the lowest values and the highest with the minimum ram usage. BTW most of the time there will be a predominance of one of the levels Highs probably.

Any ideas?

Thanks Tim

[trastikata]

Any ideas?

Code Select Expand

Dim wAverageH As Word   'Accumulator for the Highs
Dim wAverageL As Word   'Accumulator for the Lows
Dim wCurrent As Word    'Holds the current ADC readings

Main:
    'Some code to initialize the wAverageH and wAverageL from the first 2 values values as min and max

     If wCurrent > (wAverageH + wAverageL) / 2 Then
        wAverageH = (wAverageH + wCurrent) / 2   
     Else
        wAverageL = (wAverageL + wCurrent) / 2
     EndIf


[TimB]

Thanks trastikata I will check it out. In the mean time I found some data I graphed that shows what I'm working with

[TimB]

In reply to trastikata

The issue is knowing what is the high and lows to initialise it with are

Looking at the data it may be just a simple average buffer as much data as I can then add X to it for the threshold. If say on start up the value is high and it give a wrong state given a little time when things settle down it will recalibrate.

However to make it work well you want the cal refresh to be often but then if it gets a lot of highs if may give out bad results for a while until it settles again.

I'm thinking perhaps setting a factory level. Then get a long term average say hours or even days then update the value to eeprom and starting up it will be in the ballpark. Unless its not been used for months and is full of algae.

In developing this system the threshold values have been the biggest pain. You test it on the bench but on site it changes due to the fluid changing. It works again on changing manually.

Actually I think I have a solution that is that I have other stimuli that tell me that there has been consistent fluid. I can use that to say during the previous period the average will be the lowest value so just add X to it and I will be in the right area. Update the DAC to adjust the threshold and save periodically to Eeprom

[trastikata]

Tim, if there's an EEPROM in this chip you can organize the ring buffer in the EEPROM. 


You can run a RAM buffer of 15 word values and every hour you record the minimum and the maximum value from the RAM buffer in the EEPROM but each time on a different place.

For that you will need 12 extra EEPROM bytes to use as a counter for the current min and max position - 2 words as counters for the position of the last write for min and max and tripled record of each counter in case of power interrupt to not get it corrupted and lost.

This way you will have the entire EEPROM as a ring buffer. The minimum endurance for the EEPROM is 100 000 writes so the counter positions will last 12 years.

If you wish more than that, you can use another EEPROM position which will hold the number of writes in the min-max positional counter,  again triple buffered and as soon the write count approaches 100 000, switch to new 12 positions etc.

I am uncertain if explained it well, if you want I can put some code as example.

[TimB]

What I'm looking at to start with is:-

Every cycle during the testing phase I will add the ADC result to a dWordTally and inc a sample counter

At the end of of the testing phase I will divide the dWordTally by the wSampleCounter That will give me an average for the test Period

I think it will work

Tim

[top204]

Here is another method of simple averaging using accumulation. It also stores the lowest, mid and highest ADC values. It can be changed to be a moving average, and use the same principle for the low, mid, and high values storage.

Code Select Expand

'
'   /\\\\\\\\\
'  /\\\///////\\\
'  \/\\\     \/\\\                                                 /\\\          /\\\
'   \/\\\\\\\\\\\/        /\\\\\     /\\\\\\\\\\     /\\\\\\\\   /\\\\\\\\\\\  /\\\\\\\\\\\  /\\\\\\\\\
'    \/\\\//////\\\      /\\\///\\\  \/\\\//////    /\\\/////\\\ \////\\\////  \////\\\////  \////////\\\
'     \/\\\    \//\\\    /\\\  \//\\\ \/\\\\\\\\\\  /\\\\\\\\\\\     \/\\\         \/\\\        /\\\\\\\\\\
'      \/\\\     \//\\\  \//\\\  /\\\  \////////\\\ \//\\///////      \/\\\ /\\     \/\\\ /\\   /\\\/////\\\
'       \/\\\      \//\\\  \///\\\\\/    /\\\\\\\\\\  \//\\\\\\\\\\    \//\\\\\      \//\\\\\   \//\\\\\\\\/\\
'        \///        \///     \/////     \//////////    \//////////      \/////        \/////     \////////\//
'                                  Let's find out together what makes a PIC Tick!
'
' A simple ADC averaging demo program that also stores the lowest, mid, and highest ADC values read
' Written for the Positron8 compiler by Les Johnson.
'
    Device = 18F25K20                                   ' Tell the compiler what device to compile for
    Declare Xtal = 16                                   ' Tell the compiler what frequency the device will be operating at (in MHz)
'
' Create some variables
'
    Dim ADC_wLowestValue As Word = 1023                 ' Holds the lowest ADC value
    Dim ADC_wMidValue As Word                           ' Holds the mid and current ADC value
    Dim ADC_wHighestValue As Word = 0                   ' Holds the highest ADC value

'-------------------------------------------------------------------------------------------------------
' The main program starts here
' Read ADC values and average them. Storing the lowest, mid and highest values read from the ADC
'
Main:
    ANSELbits_ANS0 = 1                                  ' Set AN0 as analogue
    ADCON2bits_ADFM = 1                                 ' Setup for 10-bit ADC reading

    ADC_Get_Average()                                   ' \
    ADC_Get_Average()                                   ' | Perform dummy readings to setup the variables
    ADC_Get_Average()                                   ' /

    Do                                                  ' Create a loop
        ADC_Get_Average()                               ' Get the ADC average values
        HRsoutLn "Lowest=", Dec ADC_wLowestValue,       ' \
                 ": Mid=", Dec ADC_wMidValue,           ' | Transmit the low, mid and high ADC values on a serial terminal
                 " : Highest=", Dec ADC_wHighestValue   ' /
        DelayMs 100                                     ' A delay to see the values change on the serial terminal
    Loop                                                ' Do it forever

'-------------------------------------------------------------------------------------------------------
' Average an ADC reading and find the low, mid and high values
' Input     : None
' Output    : ADC_wLowestValue holds the lowest value
'           : ADC_wMidValue holds the mid (current) value
'           : ADC_wHighestValue holds the highests value
' Notes     : Uses a simple average by accumulation, then a division of the accumulated value.
'           : The accumulation count can be as many as the ADC_wMidValue variable will hold.
'             Then divide by the amount of accumulations performed.
'
Proc ADC_Get_Average()
    Dim bTimes As Byte

    ADC_wMidValue = 0                                   ' Reset the mid (current) ADC reading accumulator
    For bTimes = 3 DownTo 0                             ' \
        ADC_wMidValue = ADC_wMidValue + Adin(0)         ' | Accumulate the ADC readings
    Next                                                ' /
    ADC_wMidValue = ADC_wMidValue / 4                   ' Find the average value

    If ADC_wMidValue > ADC_wHighestValue Then           ' Is the ADC value greater than the last highest reading?
        ADC_wHighestValue = ADC_wMidValue               ' Yes. So load the highest variable with it

    ElseIf ADC_wMidValue < ADC_wLowestValue Then        ' Is the ADC value less than the last lowest reading reading?
        ADC_wLowestValue = ADC_wMidValue                ' Yes. So load the lowest variable with it
    EndIf
EndProc

On the serial terminal, it will show something like:

Code Select Expand

Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=563 : Highest=563
Lowest=563: Mid=573 : Highest=573
Lowest=563: Mid=573 : Highest=573
Lowest=563: Mid=573 : Highest=573
Lowest=563: Mid=573 : Highest=573
Lowest=563: Mid=583 : Highest=583
Lowest=563: Mid=604 : Highest=604
Lowest=563: Mid=614 : Highest=614
Lowest=563: Mid=634 : Highest=634
Lowest=563: Mid=644 : Highest=644
Lowest=563: Mid=655 : Highest=655
Lowest=563: Mid=675 : Highest=675
Lowest=563: Mid=696 : Highest=696
Lowest=563: Mid=706 : Highest=706
Lowest=563: Mid=726 : Highest=726
Lowest=563: Mid=726 : Highest=726
Lowest=563: Mid=726 : Highest=726
Lowest=563: Mid=726 : Highest=726
Lowest=563: Mid=726 : Highest=726
Lowest=563: Mid=726 : Highest=726
Lowest=563: Mid=726 : Highest=726
Lowest=563: Mid=716 : Highest=726
Lowest=563: Mid=716 : Highest=726
Lowest=563: Mid=716 : Highest=726
Lowest=563: Mid=716 : Highest=726
Lowest=563: Mid=706 : Highest=726
Lowest=563: Mid=696 : Highest=726
Lowest=563: Mid=675 : Highest=726
Lowest=563: Mid=665 : Highest=726
Lowest=563: Mid=644 : Highest=726
Lowest=563: Mid=624 : Highest=726
Lowest=563: Mid=614 : Highest=726
Lowest=563: Mid=593 : Highest=726
Lowest=563: Mid=583 : Highest=726
Lowest=563: Mid=573 : Highest=726
Lowest=542: Mid=542 : Highest=726
Lowest=501: Mid=501 : Highest=726
Lowest=471: Mid=471 : Highest=726
Lowest=419: Mid=419 : Highest=726
Lowest=358: Mid=358 : Highest=726
Lowest=321: Mid=321 : Highest=726
Lowest=266: Mid=266 : Highest=726
Lowest=205: Mid=205 : Highest=726
Lowest=154: Mid=154 : Highest=726
Lowest=113: Mid=113 : Highest=726
Lowest=51: Mid=51 : Highest=726
Lowest=31: Mid=31 : Highest=726
Lowest=31: Mid=31 : Highest=726
Lowest=31: Mid=31 : Highest=726
Lowest=31: Mid=31 : Highest=726
Lowest=31: Mid=31 : Highest=726
Lowest=31: Mid=31 : Highest=726
Lowest=31: Mid=31 : Highest=726
Lowest=31: Mid=41 : Highest=726
Lowest=31: Mid=41 : Highest=726
Lowest=31: Mid=41 : Highest=726
Lowest=31: Mid=41 : Highest=726
Lowest=31: Mid=41 : Highest=726
Lowest=31: Mid=61 : Highest=726
Lowest=31: Mid=82 : Highest=726
Lowest=31: Mid=92 : Highest=726
Lowest=31: Mid=102 : Highest=726
Lowest=31: Mid=123 : Highest=726
Lowest=31: Mid=133 : Highest=726
Lowest=31: Mid=143 : Highest=726
Lowest=31: Mid=143 : Highest=726
Lowest=31: Mid=143 : Highest=726


[top204]

Johnb wrote this, but I think it was on the wrong thread, so I've moved it to this thread for averaging. It is an interesting algorithm:

This is a very simple algorithm that requires minimal RAM

Code Select Expand

avg_accum = new_sample + avg_accum
avg_accum = avg_accum - sample_average
sample_average = avg_accum >> x
The value of X determines the level of damping.

[TimB]

I really appreciate the help very informative and many many thanks for the inputs

As will all situations other factors effect the outcome.

Looking at the chart above you can see that for most of the samples the values have a major bias low. So I opted to for a set period when I can know there is liquid passing the sensors I just added the ADC values and incremented a counter. At the end of the test period I just divided the total by the counts, that gave me an average that although weighted a bit by the high levels it is a good representation of the average low value.

I can use that from now on to set the threshold

6 bytes used

Thanks again

[SeanG_65]

AWESOME..I've been looking for a way of doing this for years. I created my own method but it was kind of average (excuse the pun) in it's performance in that it was alsways little lower than the actual average.