Combinatorics of Signals

[krema]

hi,
I am trying to combine multiple indicators with several different parameters into a single entry or exit signal.

Situation:

Let's say I want 3 indicators:

• ADX: timeperiod=14,
• PLUS_DI: timeperiod=[14,18,22]
• MINUS_DI: timeperiod=[15,19,23]

and create 3 signals with the indicators from above

• adx_above
• cross_plus
• cross_minus

Question:
Is it possible to create entry signals from all possible combinations of the 3 signals and with their different time periods?

What I now have is following

plus_timeperiod               14     18     22
minus_timeperiod              15     19     23
Date                                          
2019-12-31 00:00:00+00:00  False  False  False
2020-01-01 00:00:00+00:00  False  False  False
2020-01-02 00:00:00+00:00  False  False  False
2020-01-03 00:00:00+00:00  False  False  False
2020-01-04 00:00:00+00:00  False  False  False
...                          ...    ...    ...
2021-09-10 00:00:00+00:00  False  False  False
2021-09-11 00:00:00+00:00  False  False  False
2021-09-12 00:00:00+00:00  False  False  False
2021-09-13 00:00:00+00:00  False  False  False
2021-09-14 00:00:00+00:00  False  False  False

but what I want is following

plus_timeperiod               14     14     14    18   18   18 .........
minus_timeperiod              15     19     23    15   19   23 .........
Date                                          
2019-12-31 00:00:00+00:00  False  False  False  False  False  False
2020-01-01 00:00:00+00:00  False  False  False  False  False  False
2020-01-02 00:00:00+00:00  False  False  False  False  False  False
2020-01-03 00:00:00+00:00  False  False  False  False  False  False
2020-01-04 00:00:00+00:00  False  False  False  False  False  False
...                          ...    ...    ...
2021-09-10 00:00:00+00:00  False  False  False  False  False  False
2021-09-11 00:00:00+00:00  False  False  False  False  False  False
2021-09-12 00:00:00+00:00  False  False  False  False  False  False
2021-09-13 00:00:00+00:00  False  False  False  False  False  False
2021-09-14 00:00:00+00:00  False  False  False  False  False  False

Example:

import numpy as np
import vectorbt as vbt

price = vbt.YFData.download('BTC-USD', start='2020-01-01', interval='1d').get()

adx = vbt.talib('ADX').run(price['High'], price['Low'], price['Close'], timeperiod=14, short_name='adx')
plus_di = vbt.talib('PLUS_DI').run(price['High'], price['Low'], price['Close'], timeperiod=[14,18,22], short_name='plus')
minus_di = vbt.talib('MINUS_DI').run(price['High'], price['Low'], price['Close'], timeperiod=[15,19,23], short_name='minus')

adx_above = adx.real_above(20)
cross_plus = plus_di.real_above(minus_di.real, crossover=True)
cross_minus = minus_di.real_above(plus_di.real, crossover=True)

entries = cross_plus.vbt & adx_above.vbt
exits = cross_minus.vbt & adx_above.vbt
    
pf = vbt.Portfolio.from_signals(price['High'], price['Low'], price['Close'], entries, exits)

print(pf.total_return())
fig = pf.total_return().vbt.heatmap()
fig.show()

I hope my question/title is understandable.

[polakowo]

Hi @krema, take your three parameter arrays, pass them into itertools.product, unzip them, and pass each array to its corresponding indicator. This way, all three indicators will produce the same number of columns so you can use them in above and below methods.

[forLinDre]

Hi @polakowo,

I switched the order of my inplace outputs inside the apply function and it seems to at least run now without an error. I didn't realize how critical the order of these inputs is. Just to clarify, none of my in-place outputs are being used as parameters. My in-place outputs are simply variables used to calculate pct_change that I would like to be able to output if needed for troubleshooting. The goal of pct_change is to calculate the percent change in asset value from the highest point inside a rolling window, to the current low point in time. My goal is to have an entry signal generate when the pct_change is lower than my change_thres and when price is below a moving average.

In order to sweep through a variety of values for change_thres and change_window while still being able to generate an entry signal based on pct_change < change_thres, I made my change_thres parameter an output. This allows me to generate the following code to create my entry signals:

RMA_strat_indicator.pct_change_below(RMA_strat_indicator.change_thres) & RMA_strat_indicator.RMA1_above(RMA_strat_indicator.low)

I do now understand what you mean by "do all your comparisons (that's what I previously meant by combining) inside apply_func and return entries and exits". When I was initially trying to understand how the indicator factory was meant to work, I didn't think that signals should be generated within indicators. I thought that was the purpose for the signal factory and signal classes in general. For now I think I am satisfied with generating signals through snippets of code as the one above. If I need some additional functionality it's good to know that the indicator classes are flexible enough to create signals as well.

Now that my in-place outputs are at least running, their value isn't correct. My apply function is shown below:

@njit
def RMA_strat_apply_func_nb(open, high, low, max_pos, max_value, value_change, EWM_window1, EWM_window2, ewm1, ewm2, change_window, change_thres):
    RMA1 = vbt.indicators.nb.ma_nb(a=open, window=EWM_window1, ewm=ewm1)
    RMA2 = vbt.indicators.nb.ma_nb(a=open, window=EWM_window2, ewm=ewm2)
    pct_change = np.full_like(open, np.nan, dtype=np.float_)
    change_thres= np.full_like(open, change_thres, np.float_)
    for col in range(open.shape[1]):
        for i in range(EWM_window2, open.shape[0]):
            max_pos = vbt.nb.argmax_reduce_nb(col, high[i - change_window:i, col])
            max_value = high[i - change_window + max_pos, col]
            value_change = max_value - low[i, col]
            pct_change[i, col] = (value_change/max_value)*100
    return RMA1, RMA2, pct_change, change_thres

My indicator factory is set up below:

RMA_Strat_Indicator = vbt.IndicatorFactory(
    input_names=['open', 'high', 'low'],
    param_names=['EWM_window1', 'EWM_window2', 'ewm1', 'ewm2', 'change_window', 'change_thres'],
    output_names=['RMA1', 'RMA2', 'pct_change', 'change_thres'],
    in_output_names=['max_pos', 'max_value', 'value_change']
).from_apply_func(
    RMA_strat_apply_func_nb,
    EWM_window1=65,
    EWM_window2=200,
    ewm1=True,
    ewm2=True,
    change_window=6,
    change_thres=-0.1,
    param_product=True,
    run_unique=True,
    #per_column=True
)

I then run the indicator:

RMA_strat_indicator = ind.RMA_Strat_Indicator.run(
    open=Open_prefetch, high=High_prefetch, low=Low_prefetch,
    EWM_window1=65,
    EWM_window2=200,
    ewm1=True,
    ewm2=True,
    change_window= 6,
    change_thres=[-0.1, -0.15]
)

access my in-place output "max_pos" which is run by the function argmax_reduce_nb in my apply function:
RMA_strat_indicator.max_pos

However the results are unrealisitc:

I looked at the source code for argmax_reduce_nb and I don't quiet understand the use of "col" in:

@njit(cache=True)
def argmax_reduce_nb(col: int, a: tp.Array1d) -> int:
    """Return position of max."""
    a = np.copy(a)
    mask = np.isnan(a)
    if np.all(mask):
        raise ValueError("All-NaN slice encountered")
    a[mask] = -np.inf
    return np.argmax(a)

Am I not already accessing the the column inside my 2d array, high when i call out:

for col in range(open.shape[1]):
        for i in range(EWM_window2, open.shape[0]):
max_pos = vbt.nb.argmax_reduce_nb(col, high[i - change_window:i, col])

The source code doesn't seem to use the col input. Am I not using the argmax_reduce_nb function correclty?

Thank you so much for your continued assistance as i learn your library!

[polakowo]

@forLinDre, you need to specify default values for your in-place outputs, otherwise vectorbt will create them using np.empty(), which uses junk values from memory. Once you give vectorbt the default value for in-outputs, it will broadcast this value together with your inputs. Generally, in-outputs behave exactly as their name suggests: they are processed together with regular inputs when passed to the indicator and together with regular outputs when being returned by the indicator.

argmax_reduce_nb takes column because it usually used as a reduction function passed to reduce_nb, which iterates over arrays column by column and automatically passes the current column as argument to the reduction function. You can set this argument to 0.

[forLinDre]

@polakowo once I define default values for my in-outputs and then pull them after running my indicator, those default values are the only values that fill the columns of my indicator. I want my in-outputs to show their calculated values once I pull them after running the indicator. Am I misunderstanding the point of in-outputs and how to use them?

@njit
def RMA_strat_apply_func_nb(open, high, low, max_pos, max_value, value_change, EWM_window1, EWM_window2, ewm1, ewm2, change_window, change_thres):
    RMA1 = vbt.indicators.nb.ma_nb(a=open, window=EWM_window1, ewm=ewm1)
    RMA2 = vbt.indicators.nb.ma_nb(a=open, window=EWM_window2, ewm=ewm2)
    pct_change = np.full_like(open, np.nan, dtype=np.float_)
    change_thres = np.full_like(open, change_thres, np.float_)
    for col in range(open.shape[1]):
        buf = max(EWM_window1, EWM_window2, change_window)
        for i in range(buf, open.shape[0]):
            max_pos = vbt.nb.argmax_reduce_nb(0, high[i - change_window:i, col])
            max_value = high[i - change_window + max_pos, col]
            value_change = max_value - low[i, col]
            pct_change[i, col] = (value_change/max_value)*100
    return RMA1, RMA2, pct_change, change_thres

RMA_Strat_Indicator = vbt.IndicatorFactory(
    input_names=['open', 'high', 'low'],
    param_names=['EWM_window1', 'EWM_window2', 'ewm1', 'ewm2', 'change_window', 'change_thres'],
    output_names=['RMA1', 'RMA2', 'pct_change', 'change_thres'],
    in_output_names=['max_pos', 'max_value', 'value_change']
).from_apply_func(
    RMA_strat_apply_func_nb,
    max_pos=1,
    max_value=1,
    value_change=1,
    EWM_window1=65,
    EWM_window2=200,
    ewm1=True,
    ewm2=True,
    change_window=6,
    change_thres=-0.1,
    param_product=True,
    run_unique=True,
    #per_column=True
)

RMA_strat_indicator = ind.RMA_Strat_Indicator.run(
    open=Open_prefetch, high=High_prefetch, low=Low_prefetch,
    EWM_window1=65,
    EWM_window2=200,
    ewm1=True,
    ewm2=True,
    change_window= 6,
    change_thres=[-0.1, -0.15]
)

RMA_strat_indicator.max_pos

[polakowo]

All values are the same because you don't modify max_pos (such as max_pos[i, col] = ...). In-outputs must be modified in-place (that's why they are called in-place output arrays).

[forLinDre]

@polakowo thanks so much! that last part was too obvious :)

[krema]

Hi @forLinDre @polakowo, Thank you for your input and answers. I'm also working on my studies again. The original question was answered with using the product with itertools. This worked quite well for simple studies but now with increasing parameters I have a memory problem because of the many combinations I have to provide.

For optimization reasons I generated the signals (entries, exits) separately, removed duplicated columns and created then the combinations (using reshape, MultiIndex.from_tuples,columns.to_flat_index and reindex) to get the signals in the same shape so that I can use them with Portfolio.from_signals.

The signals after optimization and reshaping look like this:

ENTRIES [shape=(18, 8)]:

 adx_real_above                7                                          
 adx_timeperiod               13                                          
 adx2_real_above              9                       10                  
 adx2_timeperiod              15    15    15    15    15    15    15    15
 Date                                                                     
 2021-08-02 00:00:00+00:00  True  False  True  False  False  True  True  True


EXITS [shape=(18, 8)]:

 adx_real_below                7           8           7           8      
 adx_timeperiod               13          13          13          13      
 adx2_real_below              9     10    9     10    9     10    9     10
 adx2_timeperiod              15    15    15    15    15    15    15    15
 Date                                                                     
 2021-08-02 00:00:00+00:00  False  False  False  False  True  True  False  False

The use of these signals with the Portfolio.from_signals led to an error. I stopped investigating it further and thought this is a bug in the framework but maybe I need to use the mentioned from_apply_func function. The error was like this:

Click to expand the error

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
non-precise type array(pyobject, 0d, C)
During: typing of argument at /Users/krema/.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/nb.py (2342)

File "../../../.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/nb.py", line 2342:
def ls_enex_signal_func_nb(c: SignalContext,
    <source elided>
    """Get an element of direction-aware signals."""
    is_long_entry = flex_select_auto_nb(long_entries, c.i, c.col, c.flex_2d)
    ^

During: resolving callee type: type(CPUDispatcher(<function ls_enex_signal_func_nb at 0x137505940>))
During: typing of call at /Users/krema/.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/nb.py (2087)

During: resolving callee type: type(CPUDispatcher(<function ls_enex_signal_func_nb at 0x137505940>))
During: typing of call at /Users/krema/.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/nb.py (2087)

During: resolving callee type: type(CPUDispatcher(<function ls_enex_signal_func_nb at 0x137505940>))
During: typing of call at /Users/krema/.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/nb.py (2087)

During: resolving callee type: type(CPUDispatcher(<function ls_enex_signal_func_nb at 0x137505940>))
During: typing of call at /Users/krema/.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/nb.py (2087)

During: resolving callee type: type(CPUDispatcher(<function ls_enex_signal_func_nb at 0x137505940>))
During: typing of call at /Users/krema/.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/nb.py (2087)


File "../../../.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/nb.py", line 2087:
def simulate_from_signal_func_nb(target_shape: tp.Shape,
    <source elided>
                    is_long_entry, is_long_exit, is_short_entry, is_short_exit = \
                        signal_func_nb(signal_ctx, *signal_args)
                        ^

Now with further discussion, answers and ideas I will give it a try with from_apply_func and return the entries and exits as output_names. Let's see if this will work.

@polakowo

1. Is the error a bug?
2. Is there maybe already a way/function/indicator to combine entries/exits signals of different shapes to have the same shape (combinations) to use them with Portfolio.from_signals? If I were to create a new custom indicator that had the only job of combining multiple signals with different shapes so that it could be used for the from_apply_func function I probably couldn't combine them easily without having a memory problem. Perhaps there is another way or an idea to address this memory problem better.

[krema]

@polakowo Thanks for your memory suggestion. Will check this out.

About the error mentioned above. I have following shapes and dtypes:

(20, 8)
(20, 8)

adx_real_above  adx_timeperiod  adx2_real_above  adx2_timeperiod
7               13              9                15                 bool
                                                 15                 bool
                                                 15                 bool
                                                 15                 bool
                                10               15                 bool
                                                 15                 bool
                                                 15                 bool
                                                 15                 bool
dtype: object
adx_real_below  adx_timeperiod  adx2_real_below  adx2_timeperiod
7               13              9                15                 bool
                                10               15                 bool
8               13              9                15                 bool
                                10               15                 bool
7               13              9                15                 bool
                                10               15                 bool
8               13              9                15                 bool
                                10               15                 bool
dtype: object

Good hint to check how I call the Portfolio.from_signals. The error above occurred because I passed entries.vbt and exits.vbt instead of just entries and exits.

But still I am facing an error. This is the new error:

Click to expand the error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/var/folders/lq/wkpgb419775f28gyyzrbmrk40000gn/T/ipykernel_6124/888059939.py in <module>
----> 1 pf = vbt.Portfolio.from_signals(price['High'], price['Low'], price['Close'], entries, exits, freq='d', size=np.inf)
      2 

~/.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/portfolio/base.py in from_signals(cls, close, entries, exits, short_entries, short_exits, signal_func_nb, signal_args, size, size_type, price, fees, fixed_fees, slippage, min_size, max_size, reject_prob, lock_cash, allow_partial, raise_reject, log, accumulate, upon_long_conflict, upon_short_conflict, upon_dir_conflict, upon_opposite_entry, direction, val_price, open, high, low, sl_stop, sl_trail, tp_stop, stop_entry_price, stop_exit_price, upon_stop_exit, upon_stop_update, adjust_sl_func_nb, adjust_sl_args, adjust_tp_func_nb, adjust_tp_args, use_stops, init_cash, cash_sharing, call_seq, ffill_val_price, update_value, max_orders, max_logs, seed, group_by, broadcast_named_args, broadcast_kwargs, template_mapping, wrapper_kwargs, freq, attach_call_seq, **kwargs)
   2875             require_kwargs=dict(requirements='W')
   2876         ), broadcast_kwargs)
-> 2877         broadcasted_args = broadcast(*broadcastable_args.values(), **broadcast_kwargs)
   2878         broadcasted_args = dict(zip(broadcastable_args.keys(), broadcasted_args))
   2879         close = broadcasted_args['close']

~/.pyenv/versions/3.9.7/lib/python3.9/site-packages/vectorbt/base/reshape_fns.py in broadcast(to_shape, to_pd, to_frame, align_index, align_columns, index_from, columns_from, require_kwargs, keep_raw, return_meta, *args, **kwargs)
    562     # Get final shape
    563     if to_shape is None:
--> 564         to_shape = _broadcast_shape(*map(np.asarray, arr_args_2d))
    565 
    566     # Perform broadcasting

~/.pyenv/versions/3.9.7/lib/python3.9/site-packages/numpy/lib/stride_tricks.py in _broadcast_shape(*args)
    425         # use broadcasting to avoid allocating the full array
    426         b = broadcast_to(0, b.shape)
--> 427         b = np.broadcast(b, *args[pos:(pos + 31)])
    428     return b.shape
    429 

ValueError: shape mismatch: objects cannot be broadcast to a single shape

[polakowo]

Seems like you pass arrays with weird shapes to from_signals.

[huuquynh]

@krema Could you please to write the fixed sample. I'm a newbie and it is a bit harder for understanding and writing back to the code that @polakowo suggested.

Thank so much!

[huuquynh]

Ah, got it.

product([14,18,22], [15,19,23]), is it right?

[krema]

@huuquynh for example

params = list(zip(*list(product(np.arange(13,25), np.arange(11,29), np.arange(11,29)))))
adx = vbt.talib('ADX').run(... timeperiod=params[0])
plus_di = vbt.talib('PLUS_DI').run(..., timeperiod=params[1])
minus_di = vbt.talib('MINUS_DI').run(..., timeperiod=params[2])