[Feature Request] New store functionality - Backfill from a file prior to Live trade

[JaCoderX]

Prior starting live trading session, most strategies that use indicators will need some level of historical data backfill to fire up those indicators.
For exchanges that doesn't support OHLCV using 'backfill_start' is not an option.
backfill_start is also limited to the exchange ability to provide long historical data records.

therefore I propose implementing 'backfill_from', which will give the user the ability to load the historical data from a file instead of using a 'backfill_start'.

Notes:

• example for 'backfill_from' can be found on feed\ibdata.py
• if exchange doesn't support OHLCV data will be collected as ticks. question is how to stitch with history data in minute resolution?

This discussion has been open to gather ideas for future development

[kikingalgo]

i see this split into 2 issues

1. loading of historical data from the exchange + stiching the live feed
2. loading of historical data from a file + stiching the live feed.

i use indicators which require history of up to 120 candles (in any given range) so for 120 minutes TF i need 14400 (of 1m candles) - this is currently not working and breaking while loading the data

[JaCoderX]

loading of historical data from the exchange + stiching the live feed

This should be supported using backfill_start

[kikingalgo]

I have keen interest in resolving this issue.
I can assist with the testing

Please let me know

[JaCoderX]

example for 'backfill_from' can be found on feed\ibdata.py

@kikingalgo you can give writing a solution a try
this is in my backlog developments, but gonna take me a long time to start this one

[kikingalgo]

@JacobHanouna thanks - i will look at it

[kikingalgo]

One point i noticed today which is worth mentioning:

backfill_start cannot be used together with backfill_from

• i will override the backfill_start with the last timestamp from the last candle from backfill_from

if such case - should the feed err ? or accept any input and just override the backfill_start

[JaCoderX]

I'm not sure I followed.
maybe you can explain it using some code

[kikingalgo]

1. fetching the historical data - "historical" is set to true to end when finished fetching
   date is set to 2 days back
2. fetching live - using data0 as the source.

data0 = bt.feeds.CCXT(exchange=exchange, symbol=symbol, timeframe=bt.TimeFrame.Minutes, compression=1, backfill_start = two_days_ago, historical=true)

data1 = bt.feeds.CCXT(exchange=exchange, symbol=symbol, timeframe=bt.TimeFrame.Minutes, compression=1, backfill_from=data0 , backfill_start = two_days_ago)

when backfill_from is used, then backfill_start cannot be used at all.**
Two ways to handle this:

1. alert - raise critical error and alert the user - so it is clear that backfill_from cannot be used together with backfill_start
2. ignore - the backfill_start is ignored when supplied with backfill_from.

[JaCoderX]

when backfill_from is used, then backfill_start cannot be used at all.**

I need to investigate the code more to really answer your question.
can you point me to the place in the code where 'backfill_start' is actually used (can't find it in the code)

fetching live - using data0 as the source.

I like the idea of 'backfill_from' using a data object directly instead of a .csv file.
it will make it very versatile to use it like this.

backfill_from=data0 , backfill_start = two_days_ago

at the concept level, this to me look ambiguous. backfill is the method in which you fill the historical data.
the way I look at it, only one should be used. either you backfill your data from the exchange directly (backfill_start). or you load it from an outside source (backfill_from).

fetching the historical data - "historical" is set to true to end when finished fetching
date is set to 2 days back

why 'historical' need to be true? you wanna use the backfill_start to get data from the exchange and then go straight to live, no?

[kikingalgo]

@JacobHanouna

the way I look at it, only one should be used. either you backfill your data from the exchange directly (backfill_start). or you load it from an outside source (backfill_from).

I totally agree - i realised it while investigating - the backfill_from should include raw OHLCV candles only and not the data object - the raw data comes in the base compression and then resampled to the desired compression.

[sirhc78]

Hi,
I am not sure if my question is at the right place but I am trying to understand backtrader/ccxt/bitmex live behaviours.

I am running a test script : simple strat with bitmex data feed in 1 min bar and bitmex broker.
It computes a sma(period=2), waiting for this sma to be calculated and then send one single limit order.

2 things I don't understand :

1/If I load 4 bars from bitmex, without setting a broker, historical data is loading quickly and strategie run quickly.
With a broker it takes 1 min to run the strategy on 4 bars.
How it is possible ?
I mean if I need few days of backfill, it will take forever. Is it normal ?

Without broker :

time is  2018-09-05 13:01:32.134292  ## start
***** DATA NOTIF: DELAYED
time is  2018-09-05 13:01:36.403255  ## next
***** NEXT: 2018-09-05 10:58:00  6970.0 6974.0 6961.0 6974.0 8986602.0 Minute 1
MM = nan
time is  2018-09-05 13:01:36.404254  ## next
***** NEXT: 2018-09-05 10:59:00  6974.0 6985.0 6973.5 6984.5 13179866.0 Minute 2
time is  2018-09-05 13:01:36.404254  ## SEND BUY LIMIT: 1 @ 5979.00
***** NOTIFY ORDER 1
***** NOTIFY ORDER 2
time is  2018-09-05 13:01:36.405253  ## next
***** NEXT: 2018-09-05 11:00:00  6984.5 6985.0 6971.0 6973.5 12114261.0 Minute 3
time is  2018-09-05 13:01:36.406253  ## next
***** NEXT: 2018-09-05 11:01:00  6973.5 6975.0 6973.5 6974.5 3840378.0 Minute 4
***** DATA NOTIF: LIVE
time is  2018-09-05 13:02:03.238396  ## next
***** NEXT: 2018-09-05 11:02:00  6979.5 6979.0 6979.0 6979.0 1000.0 Minute 5
Shutdown requested...exiting

With bitmex broker :

time is  2018-09-05 12:59:39.956626  ## start
***** DATA NOTIF: DELAYED
time is  2018-09-05 12:59:51.000188  ## next
***** NEXT: 2018-09-05 10:56:00  6957.5 6970.0 6956.0 6963.0 17986946.0 Minute 1
MM = nan
time is  2018-09-05 13:00:01.276795  ## next
***** NEXT: 2018-09-05 10:57:00  6963.0 6970.0 6953.5 6970.0 12502898.0 Minute 2
time is  2018-09-05 13:00:08.912126  ## SEND BUY LIMIT: 1.0 @ 5966.50
***** NOTIFY ORDER 0
time is  2018-09-05 13:00:19.515527  ## next
***** NEXT: 2018-09-05 10:58:00  6970.0 6974.0 6961.0 6974.0 8986602.0 Minute 3
time is  2018-09-05 13:00:30.040758  ## next
***** NEXT: 2018-09-05 10:59:00  6974.0 6979.0 6973.5 6977.5 8315484.0 Minute 4
***** DATA NOTIF: LIVE
time is  2018-09-05 13:00:44.504102  ## next
***** NEXT: 2018-09-05 11:00:00  6984.5 6985.0 6971.0 6975.0 7609694.0 Minute 5
Shutdown requested...exiting

2/With a broker, why are orders sent to market during strategy initializing phase with delayed data ?
It should send orders only with "DATA MOTIF : Live" , no ?

Here is my code for this test :

`# -*- coding: utf-8 -*-
from datetime import datetime, timedelta
import ccxt
import backtrader as bt
import os
from pandas import bdate_range

#get CCXT LOGS
#import logging
#logging.basicConfig(level=logging.DEBUG)

class mmTest(bt.Strategy):
    params = (('pfast', 5), ('pslow', 30),)
    order = None

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.datetime(0)
        print('%s, %s' % (str(dt), txt))

    def __init__(self):
        # Keep a reference to the "close" line in the data[0] dataseries
        self.dataclose = self.datas[0].close
        self.mm = bt.ind.MovingAverageSimple(self.data.close,period=2)
        self.order = None

    def notify_order(self, order):
        print('*' * 5, "NOTIFY ORDER", order.status )
        
    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)
               
    def sendBuyOrder(self):
        if self.mm[0] >0:
            limit=self.mm[0] - 1000
            ticksize = 0.5
            limit=round(limit/ticksize)*ticksize
            self.order =self.buy(price= limit, exectype=bt.Order.Limit,transmit=True)
            print('time is ',datetime.now(),' ## SEND BUY LIMIT: {:s} @ {:.2f}'.format(str(self.order.size),limit ))
        else:
            print('MM = %f' % (self.mm[0]))
 
    def next(self):
        print('time is ',datetime.now(),' ## next')
        for data in self.datas:
            print('*' * 5, 'NEXT:', bt.num2date(data.datetime[0]), data._name, data.open[0], data.high[0],
                  data.low[0], data.close[0], data.volume[0],
                  bt.TimeFrame.getname(data._timeframe), len(data))
        if not self.order :
            self.sendBuyOrder()

try:
    print('time is ',datetime.now(),' ## start')
    cerebro = bt.Cerebro()
    feedconf = {'apiKey': '8KoTAFQZ3G1zkgRPSjj-LZi3', 'secret': ''}
    
    data = bt.feeds.CCXT(exchange="bitmex",symbol="BTC/USD",fromdate=datetime.now()+timedelta(hours = -2,minutes=-4),timeframe=bt.TimeFrame.Minutes,compression=1, config=feedconf,backfill_start=True)

    broker_config = {'apiKey': '8KoTAFQZ3G1zkgRPSjj-LZi3',
                         'secret': '**********',}
    
    broker = bt.brokers.CCXTBroker(exchange='bitmex',
                                       currency='XBt',
                                       config=broker_config)
    cerebro.setbroker(broker)
    
    cerebro.adddata(data)
    cerebro.addstrategy(mmTest)
    cerebro.run()
    
    #cerebro.plot()

except KeyboardInterrupt:
    print("Shutdown requested...exiting")
#except Exception:
#    traceback.print_exc(file=sys.stdout)
quit

[kikingalgo]

@sirhc78 in notify_data - you should consider the DELAY status - so in next you pass if the status != data.LIVE

example:

def __init__(self):
     self.pausetrade = self.data0.haslivedata()
     : 
def next(self)
       if self.pause_trade:
            return
       : 
def notify_data(self, data, status, *args, **kwargs):
    if self.data0.haslivedata():
           self.pausetrade = (status != data.LIVE)

[sirhc78]

Hi, thank you for your comment.

This is a way to manage live orders. And it is working.
I need to check how backtrader handles the state of a strategy ( current position and current orders) before starting running it. With this solution, it is just waiting for a new signal and start from a fresh state.

Is there a way to make it copy the exact current state and start running from this state?

Now there is still this problem of 10 seconds to run the strat on 1 bar when broker is active.
any idea ?