Using Hurst Bandpass Filters


#104

dmcsoft,

I have been playing with the various IIR filters in Octave and MatLab using filtfilt. Using Butterworth or other IIR filters you suggested padding the data.

I haven’t been able to identify the padlen command. Not in Octave and when I look it up in MatLab it doesn’t seem to be what you describe. You might have meant padarray, but that the default command pads the array with zeroes, which you are suggesting to avold. What would you padd the array with, or does it just need some sort of startup length of non-zero values, say 1 for instance? (see https://www.mathworks.com/help/images/ref/padarray.html or https://octave.sourceforge.io/image/function/padarray.html).

As an helper to anyone also messing with these filters in Octave or MatLab (or anywhere) I have identified a max filter order based upon a nominal input period (pnom) which I use as the center of a Butterworth filter for input.The max order is about the highest stable filter order you could use for a Butterworth filter which will keep the cutoff as steep as it can be for that nominal filter period and also maintain a transfer function of 1 over the pass band. I found it by experimentation and curve fitting over actual market data (stock prices and SP500 index).

Here’s the equation and code to calc the order.

order=2.948039 + (29.88074 - 2.948039)/(1 + (pnom/4.181786)^1.260203);
order=round(order,0);
lporder=order;
if order/2==fix(order/2)
    lporder=order+1;
end
  • BillC

#105

dmcsoft,

Is this analysis with FIR (Ormsby or otherwise) or IIR (Butterworth, etc.). Just curious what you did here as I have seen similar trail off effects as Dion using the IIR filters. Maximizing the order of the filter and still having it be stable seems to help mitigate the data trailing off somewhat, but it still isn’t “perfect”. Don’t think anything will be.

  • BillC

#106

BillC,

The composite line is made up of the Ormsby filter output, just as in Profit Magic.
1 low pass fiter, 5 nominal model bandpass filters and 4 additional filters which fill in the missing frequencies omitted from the nominal model.

Quoting Hurst:
“Figure IX-4 is an excellent example of what can be accomplished. In this case,
Items 1 through 6 on the chart are the outputs of six different band-pass fdters, so
designed that their response curves overlapped.”

The nominal model bandpass fiters do not overlap. The 4 additional fiters create the overlap.

The butterworth filters were created in Python using the scipy module. I presumed that filfilt was implemented in the same way as in Matlab.


#107

Hi Donal,

I also thought about the composite filter outputs issue as well, I suggested that any group of 6 arbitrarily chosen overlapping filters overlapped in such a way that their summed response curves equated to 1 (unity gain), then the composite of the filter output should equal this price output minus the error. The assumes we have a low pass, 4 band pass then a high pass.

Where does the error come from? The Ormsby filters themselves have a small but variable amount of error depending of the number of coefficients. Additionally when we use the ‘digital’ data spacing (eg 7 weeks in the Profit Magic 4.5 year filter) and the smooth the data points between the filter outputs back to say weekly bars, we are losing more accuracy as we are decimating the data points used in calculation then interpolating the spaced outputs back to our original resolution.

Let me demonstrate that it is possible to get the output of 6 filters with equally overlapping filter response curves to match the price very closely.

I will use Octave to demonstrate this and provide the octave functions and script to produce the plots for those who want to play along at home… :thinking:

In relation to viewing the filter response error, I haven’t figured out how to plot the frequency response from the filter coefficients to show the negative values. Hurst showed the error in the band pass ripple but also the ripple error lobes both positive and negative around 0 in the band stop areas. In octave, after taking an FFT of the filter coefficients then as the absolute (abs) of the FFT is taken to plot the frequency response, the negative error lobes in the stop band become positive.

I am not sure how to take the filter coefficients then plot the frequency response with the negative error lobes as illustrated by Hurst in profit magic? When creating a composite of all the filter frequency responses in order to check how much of the frequency spectrum if close to 1, (passes at source amplitude) then the displayed plot will have values that are higher than they should be where the negative lobes where actually positive due to the use of absolute.

The image below shows what I mean about the filter response Hurst illustrated has negative values.

If anyone knows how to plot the frequency response from the coefficients without using absolute to keep the negatives then I am all ears…

In this example I have used 6 evenly overlapping filters with only 1 bar data spacing for maximum accuracy and all using 1393 weights.

The designed frequency response would be 1 across the entire frequency band. With the Ormsby designs using the above it appears as below. Remember that visually this rippled line around 1 that is the filter bank composite is slightly erroneous (extra positive amounts) due to the negative response that are positive due to the absolute function of the FFT.

The output of these 6 filters compared to price is shown below.

I’m not sure how Hurst computed the with 1% figure, my findings were…
Sum DJIA price data points = 144904.3800
Sum Filter bank outputs = 144695.2712
The absolute difference = 209.1088
The average of the 2 totals = 144799.8256
Difference / Average *100 = 0.144 %

Below is a plot of the 6 filter outputs the low pass is plotted with the price…

Below are the same filters but using data spacing from 7 on low pass down to 2 on high-pass so are less accurate…

And here is the price vs filter output composite, visually you can observe slightly more error…

And here are the filter outputs…

In the less accurate outputs that used data spacing the accuracy calculations for comparison were …

Sum DJIA price data points = 144904.3800
Sum Filter bank outputs = 144468.6397
The absolute difference = 435.74034
The average of the 2 totals = 144686.5098
Difference / Average *100 = 0.3011%

Approx. 2 x less accurate on average …

In summary using a full spectrum filter bank of any different bandwidths does produce a composite very close to the price.
Granted with this perfect overlapping in reality i think it should match the price and it does with only a small percentage of error, even when using spaced data points,

I have not yet been able to find all 6 filter parameters that match the Profit Magic chart, however an exercise such as this at least proves there is not a great deal of error in the filtering process.

Attached are the Octave files you need to produce the above outputs …(remove .csv)

AAA_ormsby_w_calc_f.m - function to create the Ormsby filters.
AAA_Func_filter_series.m - function to apply the filter coefficients to a data array.
AA_HurstDJIA.m - main script file to run with embeded weekly djia price arrays.

AAA_ormsby_w_calc_f.m.csv (6.7 KB)

AAA_Func_filter_series.m.csv (2.4 KB)

AA_HurstDJIA.m.csv (118.6 KB)

Dion


#108

Hi Dion,

From my understanding the filter parameters have equal skirt slope and equal filter span. So the filter span of the 4.5Y filter is over 26 years as is the filter span of the 10 week.
These are the settings I’ve discovered, they can probably be tweaked a little.

4.5Y,199,7,0.85,1.25,2.05,2.45
80W,233,6,3.15,3.55,6.45,6.85
40W,349,4,6.85,7.25,10.65,11.05
20W,699,2,14.65,15.05,18.25,18.65
10W,1399,1,27.75,28.15,35.75,36.15

I hope this is of some use.

Donal


#109

Hi Bill,

I have ported the Prony method from the C++ source from the previously linked emptyloop site into octave.
I figured out how to create a square ‘A’ matrix and ‘B’ column matrix and use octave to find the linear equation solution (removing the need for the MultiplyMatrixVector and InverseMatrix functions from the original).

I get a bit lost following the polynomials and root finding bits of the code but it seems to run as the c++ version did. I attempted to find 1 and 2 frequencies but I am not sure if the code is correct for less than 3 frequencies, 1 frequency actually errors, 2 frequencies seems ok but I’m not sure if the chebychev code is correct for 2 frequencies. The error is small and as you have the sine amplitudes, frequencies and phases you can project the plot forward in time.

I also picked up a book by Claud Cleeton - The art of independent investing. Cleeton has working examples of using the Prony method for finding a single frequency and expands it to 2 frequencies, he then goes on to describe the general solution for 3 or more frequencies using the chebychev polynomials (which appears to be what is also described in PM appendix).

I was able to complete 1 and 2 frequencies examples from Cleeton and also port to octave, but I wasn’t able to complete 3 or more frequencies.

I have attached the two octave different versions, note they would need to be made in functions to be useful and callable, some functions were added inline so a single file could be used to test…

I am keen to analyse these prony methods in octave now and finish working through profit magic.

I believe William has hinted at this before, that a ‘one’ frequency Prony fit of the filter outputs might be an easier way to compute the average wavelength, I had previously tried to use crossing up and down over the 0 line and the find peaks octave method to work out the frequencies vs time and average.

AA_PronyNFreqTest.m.csv (15.4 KB)

AA_pronytest2.m.csv (27.9 KB)

Dion


#110

Thanks for sharing Dion! Nice work as usual. I’ll take a look. I had found a Matlab app that someone else had created that finds many, many modes. I also took a stab at a spreadsheet solution for a up to 3 mode Prony solution, but it doesn’t always yield usable results. I’ll dig it up and share when I get a chance.

I’m also very curious about Prony as I’ve read a lot about it. It seems like it would be a good place to start projecting cycles into the near future based upon the recent past when the frequencies and phases can be identified. I’d like to explore using it on past data and determining statistically how accurate and to what durations a projection can be accurate.

Between kids and my day job its hard to find the time to follow this passion! I’ll see if I can contribute more soon.

Bill