Skip to main content

web access - Parsing JavaScript Script from XML Trees


I've been trying to parse some script written in JavaScript while scraping a website, and am a little confused what functions are needed to get this done in Mathematica.


I imported the data using Import["url","XMLObject"]. The website I am trying to scrape is https://www.msn.com/en-us/money/stockdetails/history/fi-126.1.AAPL.NAS. I am basically trying to scrape historical data prices of certain dates, all of which are hidden in javascript. Any help would be of great appreciation!


The XML for the desired part looks like this:



XMLElement["script", {"type" -> 
   "text/javascript"}, {"\n    require([\"jqLoadTemplate\", \
\"dateFormat\", \"ChartsConfig\", \"chartUtils\", \
\"ChartModeController\", \"utils\", \"LocaleSettings\", \
\"NumberFormatter\", \"c.deferred\"], function ($, dateFormat, \
chartsConfig, chartUtils, ChartModeController, commonUtils, settings, \
numberFormatter) {\n        var configObject = \
chartsConfig[\"StockDetails\"];\n        var chartOptions = \
configObject.chartOptions || {};\n        if (\"False\" === \"True\") \
{\n            configObject.defaultDateRange = \

configObject.defaultOtcRange;\n        }\n\n        function \
loadTable(data) {\n            // Create a table\n            if \
(data != null && data[0] && data[0].seriesId && data[0].series) {\n   \
             var historicalData = [];\n                var size = \
data[0].series.length;\n                var formatString;\n           \
     switch (data[0].Ct) {\n                    case \"1D\":\n        \
            case \"1D_5M\":\n                        formatString = \
settings.shortTimePattern;\n                        break;\n          \
          case \"5D\":\n                        formatString = \
settings.shortTimePattern + \" \" + settings.dayDatePattern;\n        \

                break;\n                    default:\n                \
        formatString = settings.monthDayYearPattern;\n                \
}\n\n                var settingsCopy = $.extend(true, {}, settings);\
\n                settingsCopy.numberDecimalDigits = 0;\n\n           \
     for (var j in data[0].series) {\n                    i = \
data[0].series[size - 1 - j];\n                    var date = \
dateFormat(chartUtils.normalizeDate(i[0]), formatString);\n           \
         var open = numberFormatter.formatNumber(i[4], settings);\n   \
                 var close = numberFormatter.formatNumber(i[1], \
settings);\n                    var low = \

numberFormatter.formatNumber(i[6], settings);\n                    \
var high = numberFormatter.formatNumber(i[5], settings);\n            \
        var volume = numberFormatter.formatNumber(i[3], \
settingsCopy);\n                    historicalData.push({\n           \
             \"date\": date,\n                        \"high\": high,\
\n                        \"low\": low,\n                        \
\"open\": open,\n                        \"close\": close,\n          \
              \"vol\": volume\n                    });\n              \
  }\n                \
$('#containerTable').loadTemplate(('#trTemplate'), historicalData);\n \

           }\n\n        }\n        //dataUrl is URL template. \
urlResolver inserts the parameters.\n        function \
onModeChanged(timeRangeSelected, chartTypeSelected) {\n            \
var dataUrl = configObject.getUrl(timeRangeSelected, \"False\", \
\"False\");\n            var urlResolver = configObject.urlResolver;\n\
            var finalUrl = urlResolver(dataUrl, { symbol: \
\"126.1.AAPL.NAS\", isEOD: \"False\", locale: \"en-US\", \
timeRangeSelected: timeRangeSelected, chartTypeSelected: \"ohlc\", \
isVolumeChartSupported: true });\n            \
chartUtils.getData(finalUrl, configObject.dataBuilder, loadTable, \

function () { }, \"ohlc\");\n        }\n        var timeRange = \
configObject.getTimeRange(\"False\", \"False\", \"126.1.AAPL.NAS\");\n\
            var control = \
ChartModeController.initialize(configObject, onModeChanged, \
\"False\", {}, \"126.1.AAPL.NAS\", timeRange);\n            var \
modeSelector = \
document.getElementById(\"heropanechart-mode-selector\");\n        \
modeSelector.appendChild(control);\n        onModeChanged(timeRange, \
configObject.getDefaultChartType());\n    });\n    "}], "\n ",

Answer




You have misunderstood how this works. What happens is that JavaScript calls a server-side script which returns the data, and the data is subsequently inserted into the HTML. The data cannot be found in the HTML document (in the HTML or in the JavaScript) to begin with.


The most straightforward way to get the data is to monitor the network and observe what URLs the JavaScript code is calling. There is a network tab in Chrome's web developer tool that will do this for you. I opened up the network tab, clicked the "WEEK" tab on the website and took notice of what URL was added to the list of called URLs. It turned out to be this one:


url = "https://finance.services.appex.bing.com/Market.svc/ChartDataV5?symbols=126.1.AAPL.NAS&chartType=5d&isEOD=False&lang=en-US&isCS=true&isVol=true";

The format of the data is JSON. We can import it like this:


data = Import[url, "JSON"];
values = "Series" /. data;

values is now a list of elements such as this:


{"T" -> 10260, "P" -> 141.23, "Hp" -> 141.25, "Lp" -> 141.14, 

 "Op" -> 141.14, "V" -> 34837}

It is unclear what timestamp format is being used, but Hp is definitely the high, Lp the low, Op the open value, p the close value, and V the volume.


To get data for other stocks you have to make the appropriate update to the URL. Changing AAPL.NAS (Apple) to MSFT.NAS (Microsoft) seems to work.


Of course, this is not how the website is meant to be used. They may change the URLs in whatever way they chose, at any moment in time. But this approach may very well work for downloading the data that you want.


Comments

Post a Comment

Popular posts from this blog

plotting - Plot 4D data with color as 4th dimension

I have a list of 4D data (x position, y position, amplitude, wavelength). I want to plot x, y, and amplitude on a 3D plot and have the color of the points correspond to the wavelength. I have seen many examples using functions to define color but my wavelength cannot be expressed by an analytic function. Is there a simple way to do this? Answer Here a another possible way to visualize 4D data: data = Flatten[Table[{x, y, x^2 + y^2, Sin[x - y]}, {x, -Pi, Pi,Pi/10}, {y,-Pi,Pi, Pi/10}], 1]; You can use the function Point along with VertexColors . Now the points are places using the first three elements and the color is determined by the fourth. In this case I used Hue, but you can use whatever you prefer. Graphics3D[ Point[data[[All, 1 ;; 3]], VertexColors -> Hue /@ data[[All, 4]]], Axes -> True, BoxRatios -> {1, 1, 1/GoldenRatio}]
Post a Comment
Read more

plotting - Filling between two spheres in SphericalPlot3D

Manipulate[ SphericalPlot3D[{1, 2 - n}, {θ, 0, Pi}, {ϕ, 0, 1.5 Pi}, Mesh -> None, PlotPoints -> 15, PlotRange -> {-2.2, 2.2}], {n, 0, 1}] I cant' seem to be able to make a filling between two spheres. I've already tried the obvious Filling -> {1 -> {2}} but Mathematica doesn't seem to like that option. Is there any easy way around this or ... Answer There is no built-in filling in SphericalPlot3D . One option is to use ParametricPlot3D to draw the surfaces between the two shells: Manipulate[ Show[SphericalPlot3D[{1, 2 - n}, {θ, 0, Pi}, {ϕ, 0, 1.5 Pi}, PlotPoints -> 15, PlotRange -> {-2.2, 2.2}], ParametricPlot3D[{ r {Sin[t] Cos[1.5 Pi], Sin[t] Sin[1.5 Pi], Cos[t]}, r {Sin[t] Cos[0 Pi], Sin[t] Sin[0 Pi], Cos[t]}}, {r, 1, 2 - n}, {t, 0, Pi}, PlotStyle -> Yellow, Mesh -> {2, 15}]], {n, 0, 1}]
Post a Comment
Read more

plotting - Mathematica: 3D plot based on combined 2D graphs

I have several sigmoidal fits to 3 different datasets, with mean fit predictions plus the 95% confidence limits (not symmetrical around the mean) and the actual data. I would now like to show these different 2D plots projected in 3D as in but then using proper perspective. In the link here they give some solutions to combine the plots using isometric perspective, but I would like to use proper 3 point perspective. Any thoughts? Also any way to show the mean points per time point for each series plus or minus the standard error on the mean would be cool too, either using points+vertical bars, or using spheres plus tubes. Below are some test data and the fit function I am using. Note that I am working on a logit(proportion) scale and that the final vertical scale is Log10(percentage). (* some test data *) data = Table[Null, {i, 4}]; data[[1]] = {{1, -5.8}, {2, -5.4}, {3, -0.8}, {4, -0.2}, {5, 4.6}, {1, -6.4}, {2, -5.6}, {3, -0.7}, {4, 0.04}, {5, 1.0}, {1, -6.8}, {2, -4.7}, {3, -1....
Post a Comment
Read more