Using Hurst Bandpass Filters

Hi Curt,

Thanks for the compliments. I feel that since the cycle periods do slowly vary, but are fairly consistent and we really don’t (and probably can’t) accurately predict the composite price into the future a way to get an a good idea of when a nest of cycles will occur is using statistics. So that’s where the statistical summary arises. I look for a “future cycle trough zone” using an average cyle and standard deviation with a 95% confidence interval (about 1.645x the s.d.) to define the width of the expected trough zone.
That’'s why I started those summaries. It seemed an obvious way to me to take advantage of the pseudo-normal filter lengths.

I agree Fourier is somewhat unreliable. I’m not even sure how much to trust the Lanczos method either. Currently I only have it programmed into VB/OpenOffice. I think I’ll try to create an Octave version as Dion has done to test how it compares to the built in versions. Zero padding seems to create certain artifacts in the FFT method built into Octave/Matlab similar to the ringing effect in bandpass filters outside of the cutoff range. What do you typically use for your Fourier length? I’ve been using anywhere from 4096 to 8000 data points.

In terms of calendar vs trading days. I’ve found that adjusting for the effect of the difference between the two is fairly effective using scale factors. For instance there’s approximately 4.8155 trading days per calendar week on average. This is a about 0.69 times a calendar period so it works out to something like this in terms of Hurst’s nominal model…

% Frame     18y     9y      4.5y    80w     40w     20w     10w     5w      20d     10d     5d
% Monthly   215.10  107.55  53.78   17.93   8.96    4.48    2.24    1.12    0.54    0.28    0.14
% Weekly    935.31  467.66  233.83  77.94   38.97   19.49   9.74    4.87    2.36    1.23    0.62
% Daily     6547.2  3273.60 1636.80 545.60  272.80  136.40  68.20   34.10   16.50   8.60    4.31
% Trd Day   4504.01 2252.00 1126.00 375.33  187.67  93.83   46.92   23.46   11.35   5.92    2.96

This seems to work out well, although if you are using timeframes shorter than 5 weeks or so I think the error could be significant.

Curious how you use the Fourier analysis to determine your filter cutoffs. My thoughts were use the troughs in the Fourier to create a bandpass filter to pick up the cycle between them. It doesn’t seem to work so well all the time and the “right” troughs are not always obvious. I also am unsure how to select the “dominant” wave.

Good discussion. Thanks.

  • BillC
1 Like

Curt,

I created a calendar day interpolator to show you a comparison of trading day versus calendar day analysis. Here’s an example of the 10 week cycle using the trading day to calendar day scaling that I mentioned above. I think the difference is fairly negligible when doing an analysis. You just need to make sure you account for cd’s vs td’s in any coding and analysis. Pink is in trading days (a 46.92+/-25% filter) while the black is in calendar days (68.2+/-25%) filter. Both are Butterworth filters applied with filtfilt. If you recall the scaling factor I noted was 0.69: 46.92/68.2 = 0.688 (~0.69).

  • Bill C
1 Like

Hi Bill,

I am also at a loss as to how to determine how to tie all the spectral analysis methods together to come up with methods to determine the cycles bands that should be extracted (i.e. a model).

Using fourier on the filter output to determine the order of smaller cycles that may be present within the filtered range is an interesting idea.

Is the y scale of the filter output in you last post correct (the largest cycle has a amplitude of around 0.05)?

My initial Excel VBA attempt at Lanczos method was slow and after porting to Octave it was also slow (over a minute to run the DJIA data for the chart like in PM). I found the issue was two nested for loops for the sin cos calcs, I guess even 1000 data points * 1000 is 1 Million iterations. I recently converted the Octave code so its nearly all vectorized code now removing nearly all the for loops and it runs in under a second now.

I have attached this attempt at Lanczos method in Octave - (remove .csv extention for octave).

LanczosExample.m.csv (6.0 KB)

Dion

Dion,

Thanks for the code. That’ll save me a little bit of time! That amplitude is .05, but I think I probably used the filter on log(price) rather than raw price. Up a little higher I posted a picture of amplitude and phase using Octave’s built in Fourier function and it shows an amplitude more like 20ish for the 10w filter, which I’m sure uses the raw price. (see my January 13th post)

I had an idea to produce a periodogram using this idea - Fourier of the filter output - that would combine the comb filter with Fourier anakysis. Basically perform a Fourier of successive overlapping comb’s. The Fourier of the filter output looks so smooth (Jan 13th post) and my research indicates that picking the peaks within the cutoff usually reproduces most of the filter, so I wonder if doing Fourier’s of the comb’s would make a smoother periodogram to help develop the nominal model?

Good to see this conversation rolling again!

  • BillC

Hi Bill,

Using statistics to help identify price waves is also called ensemble averaging. Below is an example. The chart on the left is a single measurement of a noisy signal. The chart on the right is an average of 9 repeated measurements of the same signal. I used to think using averages would produce average trading results but I have completely changed my view on this after digging into the Fourier series and doing some crude backtesting. I need all the help I can get.

image

Below is a link that explains the issues with FFT far better than I could. It also provides some general guidelines for appropriate sample sizes and starting points for both FFT and DFT.

https://www.dataq.com/data-acquisition/general-education-tutorials/fft-fast-fourier-transform-waveform-analysis.html

Your question about sample size is a good one. Hurst used 1,144 data points (44 years times 52 wks/yr divided by 2) to do his Fourier analysis in PM. On the other hand, he used only three oscillations of the 100 calendar day wave (300 data points) when using a periodogram for his trading analysis in the course material. I think the answer depends on what you are trying to accomplish. To derive a nominal model, you need a lot of data, and ideally, it should be windowed over multiple overlapping time periods (e.g. p.198 in PM). Hurst showed comb filters provide the same information as a Fourier analysis with higher resolution. My limited experience is that the Fourier series method described in PM works just as well on shorter time periods while minimizing the spectral leakage and wrapping (end-point mismatch) issues found when using FFT.

Have you and Dion looked at the book “Applied Analysis” by Lanczos in the references to PM? This book is a beast to read, but it explains a number of the issues we have been struggling with in this thread including giving a few numerical examples, which I always find helpful. In one example, Lanczos described how to empirically determine filter cutoffs from a Fourier series with 67 data points. The two chapters I found most helpful were Harmonic Analysis (IV) and Data Analysis (V). BTW, Lanczos was a research assistant for Albert Einstein at one point in his career, so he has some serious street cred.

I agree with your points on calendar versus trading days. I have been attempting to shorten up my trading time frame (5d, 10d, and 20d nominal waves) to take advantage of increased compounding and have personally found that calendar days are more accurate from that perspective.

I think the difference between our 10w waves is the fact we are using different nominal models. Since I was doing all this work anyway, I figured rather than build filters based on what Hurst did 50 yrs ago, I’d try to determine for myself what the correct nominal model is using very long (approximately 220 yrs), long (100 yrs), intermediate (50 yrs) and short (25 yrs) periods of time. The results are very stable and confirm Hurst’s comment that the spectral lines drift slowly over time. Keep in mind, 50 yrs is still a long time so his results in PM are not exactly the same as what you find today. The differences are obviously smaller on the short waves but not insignificant when trading. Appropriate filter cutoffs should accommodate for this spectral drift.

The “dominant wave” is best left for another post. Rather than reinvent the wheel (believe me, I have tried), the high pass filter output (inverse MA in PM) seems to work just fine. Hurst also discussed wave dominance in the course material when explaining his periodogram analysis, but I personally think it is articulated more clearly in PM.

Curt

Bill and Dion,

Another book in the references of PM that looks promising is Milne’s “Numerical Calculus”. There are sections on trigonometric (Fourier) least squares approximation techniques and harmonic analysis. Most of these approaches assume equal spaced data so calendar days should be used. Just digging into it now.

Curt

Dion,

I’ve been thinking about your questions (and mine) about development of the Nominal model the way that Hurst presented it in PM.

So what does it all mean, I’m really not sure, spectral analysis was used to come up with a detailed nominal model in which periods were separated based on troughs in Fourier, then we are shown a chart with filter outputs that throws the separations out the window for the filters used … why? Why combine 3 year and 4.5 and 18 month and 12 month, and skip the 26 and 13 week filters? Was it because of observational periods more frequently matched in a range of the combined periods or the combined filter outputs ‘looked’ better? Do the filter frequencies align with the troughs and the 0.3676 minimum frequency spans? Not exactly in my findings. Do they appear to be exact harmonic multiples of each other? Not in my findings either. I haven’t been able to understand the reasoning behind all of these anomalies yet.

I’m thinking some of this must have been done to select a model that is harmonic to help in analyzing and trading. There are so many different possibilities for frequencies, but using them all just isn’t practical. Selecting a model that is nearly (or completely harmonic) limits the number of waves to track and gives a basis for finding “likely nests of lows”. in other words Hurst wasn’t trying to make a “complete” model, maybe he was trying to simply make a practical one that could more easily be handled and that explains a great deal of price action. So skipping some of the cycles and combining others was a strategy, not really pure science. Its a practical simplification based on science. As a design engineer that is something that I can understand and get behind.

So when developing a nominal model look for strong cycles or groups of cycles that can be combined in sense where they are harmonically related and synchronized. Don’t try to use all of them. The average nature over time and the principals of harmonicity and synchronicity should work out often over a long period of time.

Curious what others may think of this idea.

  • BillC

Hi Curt,

I haven’t read Lanczos’ book, “Applied Analysis”, yet. I have scanned through the TOC and found it to be potentially intriguing. Might be a good idea to pick up a copy or borrow one fr the Fourier analysis example you mention. I’m an engineer and had already known about Lanczos from my school days, so yeah, he’s got cred! One of those famous mathematician types.

I’m still puzzling through how to really construct a “valid” Nominal model although I think I’m getting close to a process. If I have some success I’ll try to explain hw I would go about it. I’m trying to mainly use the spectral approach and practical techniques, but I’m sure there will end up being some amount of judgement and artistry to it as well.

From my current perspective it’s a bit of 1) spectral evidence, 2) harmonicity, 3) persistence and 4) synchronicity.mixed together.

Thanks for the tips.

  • BillC

Hi Bill,

The Lanczos book is a classic and I now believe it was the foundation for most of Hurst’s work. It was a real eye opener for me. I’m guessing it will answer many of your questions above. The great thing about the old books in the bibliography of PM is that they are cheap and the payback is large. Plus, the terminology is consistent with what Hurst used. This was important to me because I am not an engineer. I work in finance and had to unlearn 30+ years of training before I found success using Hurst’s spectral approach.

The mathematical principles that Hurst based his spectral approach on need to be fully understood and then applied and interpreted in a practical way to real-life market situations. This is no small task, BTW. It requires intense study and then practice along with some backtesting. By backtesting, I mean finding waves to trade that consistently make you money, not developing a systematic or automated trading strategy. Hurst himself said the key to success using his spectral approach is the interpretation of ambiguities in the filter output.

Curt

Thanks Curt. I’ll check out the Lanczos books. Ordered a paperback from Amazon.

  • BillC