Problem
We can have data at some location in csv, and we can write some scripts which are scheduled to get the data via FTP at the end of the day. Where do we store the data? How do we store the data in such a way that we can do various analyses of it?
Proposal #1 - Keep using unorganized input file
Proposal #2 - Use a database
Advantages:
Proposal #3 - Show simply as xml or json if requirement is minimalIf you were integrating a feed of end of day stock price information (open, high, low, and closing price) for 5,000 companies, how would you do it? You are responsible for the development, rollout and ongoing monitoring and maintenance of the feed. Describe the different methods you considered and why you would recommend your approach. The feed is delivered once per trading day in a comma-separated format via an FTP site. The feed will be used by 1000 daily users in a web application.Solution
We can have data at some location in csv, and we can write some scripts which are scheduled to get the data via FTP at the end of the day. Where do we store the data? How do we store the data in such a way that we can do various analyses of it?
Proposal #1 - Keep using unorganized input file
- Keep the data in text files. This would be very difficult to manage and update, as well as very hard to query. Keeping unorganized text files would lead to a very inefficient data model.
Proposal #2 - Use a database
- We could have some code in perl (as it processes string's pretty well) or some other language, which inserts data in the database, and preferably bulk update.
- Historical cleanup of records may be required
- Logical storage of data.
- Facilitates an easy way of doing query processing over the data.
Advantages:
- Makes the maintenance easy once installed properly.
- Roll back, backing up data, and security could be provided using standard database features. We don’t have to “reinvent the wheel.”
<Stocks> <date value=“2008-10-12”> <company name=“foo”> <open>126.23</open> <high>130.27</high> <low>122.83</low> <closingPrice>127.30</closingPrice> </company> <company name=“bar”> <open>52.73</open> <high>60.27</high> <low>50.29</low> <closingPrice>54.91</closingPrice> </company> </date> <date value=“2008-10-11”> . . . </date> </Stocks>
References
http://tianrunhe.wordpress.com/2012/04/08/design-a-feed-system-of-stock-informaiton/
0 comments:
Post a Comment