|
Extract useful content from a HTML page using Go |
|
I started to learn Go (golang) programming language. As a first problem i decided to code "Hello, world!" a tool to extract useful content from a HTML page.
How does it work.
I use the goquery package to convert a HTML page to well structured DOM tree.
Next step is to drop all nodes (tags) from a HTML page that are not useful at all. Like, script, style, head etc.
Then for each mode of a tree (starting from a top level node) i repeat a function recursively. The function detects if a node has subnodes.
If yes, then the function checks how a text is distributed between child nodes of a node.
If a text is distributed relatively evenly, then current node is what we are looking for and a text from it is returned as a result.
If a text is gathered mainly in one of subnodes, then a function is executed for this subnode recursively.
To detect if a text is distributed evenly to get total length of all text (ignoring HTML) in a node and then calculate a Mean Deviation of lengths of a text in each subnode. Having this 2 values i am checking if a Mean Deviation is relatively small to a full text length. If yes , then the text is distributed relatively evenly.
There is the code! https://github.com/Gelembjuk/articletext
Usage:
package main
import (
"fmt"
"os"
"github.com/gelembjuk/articletext"
)
func main() {
url := os.Args[1]
text, err := articletext.GetArticleTextFromUrl(url)
fmt.Println(text)
}
|
|
Last Updated on Wednesday, 06 July 2016 17:11 |
|
|
Implementing web site sign-up using popular social networks login in PHP |
|
Majority of web sites use sign-up with social networks as alternate for traditional registration/login. Buttons “Login with: Facebook, Twitter, Google,... ” are present together with a login form very often now.
Using social networks for login simplifies a user experience and saves his time.
The author used social login on different web sites as well. Based on experience it was decided to build a PHP class/package that summarise the social login process on an abstract level and allows to implement the process with minimum coding.
Social login in common
For a user a social login looks like identical process in different social networks. Usually, it looks like: a user licks on a “login with ...” button. Is redirected to a social network website (http://facebook.com etc.) , clicks “Allow” on that site and, finally, is redirected back to an original website where his social login is processes and a session for a user is created.
However, on a back-end there is some difference.
Some social networks APIs don't need to connect to an api endpoint before to redirect a user a login page. Other has to connect to an API to init a login process, to get some special redirect url, to check some settings etc.
Next difference is that some APIs will require saving a specific data (request token etc) between 2 steps of an auth process – get a redirect url and complete login. For others no need to save something.
All such differences must be taken into account during a social login process.
Introduction gelembjuk/auth PHP package to unify the social login process.
The package is available in Composer. It allows to login with Facebook, Twitter, Google, LinkedIn with unified process and minimum coding. Additionally, the package can be easy extended with new social networks and same process will be used.
Installation
It is just normal installation with Composer
composer require gelembjuk/auth
|
|
Last Updated on Thursday, 30 July 2015 04:38 |
|
Sharing my experience in PHP programming. Error caching, Social login, Templaters, Email formatting/sending |
|
After 13 years of PHP programming I decided to share some experience . In my Github profile https://github.com/Gelembjuk i uploaded couple my classes/packages which can be useful for other developers.
For now I have shared 4 packages. All are also available with the Composer (https://packagist.org/) and are very easy to install (as any package available in composer listings)
Logger
gelembjuk/logger in composer, https://github.com/Gelembjuk/logger in Github.
The package contains classes for logging, error catching and display.
ErrorScreen class helps to catch different types of errors, PHP Exceptions, Warnings and Fatal errors, log error information and display correct error screen to a user depending on expected formats (html, json, xml, http).
FileLogger class based on Psr/Log helps to filter what to log. You can set On/Off different log groups. The main idea of this class is to help to debug big applications. But it would also work as a normal "everyday" logger to a file. It is used in my other packages/classes but can be replaced with any other Psr/Log logger.
ApplicationLogger trait helps to add logging functionality in different classes very easy. You don't need to care about such methods like log(),debug(), setLogger(), getLogger() etc. In your classes when this trait is used. Is really useful big big applications with many classes .
|
|
Last Updated on Wednesday, 29 July 2015 18:06 |
|
Business opportunities around MOOC |
|
MOOC (massive open online courses) become popular more and more. Lots of people are involved in a discussion about MOOC. E-Learning portals publish news and press releases related to MOOC every day. Millions of students learn courses and leave their feedback. One of most popular discussions related to MOOCs is about if is this possible MOOCs replace traditional higher education.
Many experts think it is too early to say about real revolution in higher education as MOOC can not ensure getting of all skills a student can get in a college. A practice aspect of education process is missed. And also a college atmosphere is very important thing and can not be compensated with something when a student learns with MOOCs.
MOOC providers don't have alternative for things outlined above. But someone other can provide alternatives to that things. And this is good business opportunity to offer services around MOOCs aimed to replace traditional college things missed in MOOCs.
|
|
Last Updated on Saturday, 16 March 2013 06:12 |
|
Future of higher education |
|
By Roman Gelembjuk
Download PDF version
Will higher education system differ in future?
To get the answer on this question we need to understand if is everything all right in higher education. If modern higher education system is correct and everyone (or almost everyone) is happy with it then no need in changes. But if there are problems then we can expect changes.
Does modern higher education system have problems? If we care only about USA and West Europe (and other developed countries) then we could say “Higher Education will slowly transform under the influence of technology. All core principles will stay unchanged”. The reason is obvious. Developed countries built good higher education system. It filters gifted and talented children and allow them to get higher education for free or for low price. Such system seems even and, in general, satisfies the requirements of society. Of course, there are problems. On my opinion, on of biggest problems is not reasonable value for money . Nevertheless, imbalance is not sufficiently serious to accelerate changes.
What about developing countries? Population of such countries is bigger then in developed countries and there is no fair higher education system. Many gifted and talented children have no chance to get higher education. The have choice – pay big price to get good higher education or pay less money to get low quality education (formal higher education without real background) or get nothing and lose chance to go up with social ladder. What if they get some alternative that are cheaper then traditional higher education? They definitely need the alternative because something is better then nothing.
|
|
Last Updated on Wednesday, 07 November 2012 13:31 |
|
|
|
|
<< Start < Prev 1 2 3 4 5 6 Next > End >>
|
|
Page 1 of 6 |
