Skip to main content

web access - Parsing JavaScript Script from XML Trees


I've been trying to parse some script written in JavaScript while scraping a website, and am a little confused what functions are needed to get this done in Mathematica.


I imported the data using Import["url","XMLObject"]. The website I am trying to scrape is https://www.msn.com/en-us/money/stockdetails/history/fi-126.1.AAPL.NAS. I am basically trying to scrape historical data prices of certain dates, all of which are hidden in javascript. Any help would be of great appreciation!


The XML for the desired part looks like this:



XMLElement["script", {"type" -> 
   "text/javascript"}, {"\n    require([\"jqLoadTemplate\", \
\"dateFormat\", \"ChartsConfig\", \"chartUtils\", \
\"ChartModeController\", \"utils\", \"LocaleSettings\", \
\"NumberFormatter\", \"c.deferred\"], function ($, dateFormat, \
chartsConfig, chartUtils, ChartModeController, commonUtils, settings, \
numberFormatter) {\n        var configObject = \
chartsConfig[\"StockDetails\"];\n        var chartOptions = \
configObject.chartOptions || {};\n        if (\"False\" === \"True\") \
{\n            configObject.defaultDateRange = \

configObject.defaultOtcRange;\n        }\n\n        function \
loadTable(data) {\n            // Create a table\n            if \
(data != null && data[0] && data[0].seriesId && data[0].series) {\n   \
             var historicalData = [];\n                var size = \
data[0].series.length;\n                var formatString;\n           \
     switch (data[0].Ct) {\n                    case \"1D\":\n        \
            case \"1D_5M\":\n                        formatString = \
settings.shortTimePattern;\n                        break;\n          \
          case \"5D\":\n                        formatString = \
settings.shortTimePattern + \" \" + settings.dayDatePattern;\n        \

                break;\n                    default:\n                \
        formatString = settings.monthDayYearPattern;\n                \
}\n\n                var settingsCopy = $.extend(true, {}, settings);\
\n                settingsCopy.numberDecimalDigits = 0;\n\n           \
     for (var j in data[0].series) {\n                    i = \
data[0].series[size - 1 - j];\n                    var date = \
dateFormat(chartUtils.normalizeDate(i[0]), formatString);\n           \
         var open = numberFormatter.formatNumber(i[4], settings);\n   \
                 var close = numberFormatter.formatNumber(i[1], \
settings);\n                    var low = \

numberFormatter.formatNumber(i[6], settings);\n                    \
var high = numberFormatter.formatNumber(i[5], settings);\n            \
        var volume = numberFormatter.formatNumber(i[3], \
settingsCopy);\n                    historicalData.push({\n           \
             \"date\": date,\n                        \"high\": high,\
\n                        \"low\": low,\n                        \
\"open\": open,\n                        \"close\": close,\n          \
              \"vol\": volume\n                    });\n              \
  }\n                \
$('#containerTable').loadTemplate(('#trTemplate'), historicalData);\n \

           }\n\n        }\n        //dataUrl is URL template. \
urlResolver inserts the parameters.\n        function \
onModeChanged(timeRangeSelected, chartTypeSelected) {\n            \
var dataUrl = configObject.getUrl(timeRangeSelected, \"False\", \
\"False\");\n            var urlResolver = configObject.urlResolver;\n\
            var finalUrl = urlResolver(dataUrl, { symbol: \
\"126.1.AAPL.NAS\", isEOD: \"False\", locale: \"en-US\", \
timeRangeSelected: timeRangeSelected, chartTypeSelected: \"ohlc\", \
isVolumeChartSupported: true });\n            \
chartUtils.getData(finalUrl, configObject.dataBuilder, loadTable, \

function () { }, \"ohlc\");\n        }\n        var timeRange = \
configObject.getTimeRange(\"False\", \"False\", \"126.1.AAPL.NAS\");\n\
            var control = \
ChartModeController.initialize(configObject, onModeChanged, \
\"False\", {}, \"126.1.AAPL.NAS\", timeRange);\n            var \
modeSelector = \
document.getElementById(\"heropanechart-mode-selector\");\n        \
modeSelector.appendChild(control);\n        onModeChanged(timeRange, \
configObject.getDefaultChartType());\n    });\n    "}], "\n ",

Answer




You have misunderstood how this works. What happens is that JavaScript calls a server-side script which returns the data, and the data is subsequently inserted into the HTML. The data cannot be found in the HTML document (in the HTML or in the JavaScript) to begin with.


The most straightforward way to get the data is to monitor the network and observe what URLs the JavaScript code is calling. There is a network tab in Chrome's web developer tool that will do this for you. I opened up the network tab, clicked the "WEEK" tab on the website and took notice of what URL was added to the list of called URLs. It turned out to be this one:


url = "https://finance.services.appex.bing.com/Market.svc/ChartDataV5?symbols=126.1.AAPL.NAS&chartType=5d&isEOD=False&lang=en-US&isCS=true&isVol=true";

The format of the data is JSON. We can import it like this:


data = Import[url, "JSON"];
values = "Series" /. data;

values is now a list of elements such as this:


{"T" -> 10260, "P" -> 141.23, "Hp" -> 141.25, "Lp" -> 141.14, 

 "Op" -> 141.14, "V" -> 34837}

It is unclear what timestamp format is being used, but Hp is definitely the high, Lp the low, Op the open value, p the close value, and V the volume.


To get data for other stocks you have to make the appropriate update to the URL. Changing AAPL.NAS (Apple) to MSFT.NAS (Microsoft) seems to work.


Of course, this is not how the website is meant to be used. They may change the URLs in whatever way they chose, at any moment in time. But this approach may very well work for downloading the data that you want.


Comments

Post a Comment

Popular posts from this blog

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...
Post a Comment
Read more

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...
Post a Comment
Read more

dynamic - How can I make a clickable ArrayPlot that returns input?

I would like to create a dynamic ArrayPlot so that the rectangles, when clicked, provide the input. Can I use ArrayPlot for this? Or is there something else I should have to use? Answer ArrayPlot is much more than just a simple array like Grid : it represents a ranged 2D dataset, and its visualization can be finetuned by options like DataReversed and DataRange . These features make it quite complicated to reproduce the same layout and order with Grid . Here I offer AnnotatedArrayPlot which comes in handy when your dataset is more than just a flat 2D array. The dynamic interface allows highlighting individual cells and possibly interacting with them. AnnotatedArrayPlot works the same way as ArrayPlot and accepts the same options plus Enabled , HighlightCoordinates , HighlightStyle and HighlightElementFunction . data = {{Missing["HasSomeMoreData"], GrayLevel[ 1], {RGBColor[0, 1, 1], RGBColor[0, 0, 1], GrayLevel[1]}, RGBColor[0, 1, 0]}, {GrayLevel[0], GrayLevel...
Post a Comment
Read more