Main Menu

Login Form

Syndication

Welcome to the Roman Gelembjuk's blog

Extract useful content from a HTML page using Go

I started to learn Go (golang) programming language. As a first problem i decided to code "Hello, world!" a tool to extract useful content from a HTML page.

How does it work.

I use the goquery package to convert a HTML page to well structured DOM tree.

Next step is to drop all nodes (tags) from a HTML page that are not useful at all. Like, script, style, head etc.

Then for each mode of a tree (starting from a top level node) i repeat a function recursively. The function detects if a node has subnodes.

If yes, then the function checks how a text is distributed between child nodes of a node.

If a text is distributed relatively evenly, then current node is what we are looking for and a text from it is returned as a result.

If a text is gathered mainly in one of subnodes, then a function is executed for this subnode recursively.

To detect if a text is distributed evenly to get total length of all text (ignoring HTML) in a node and then calculate a Mean Deviation of lengths of a text in each subnode. Having this 2 values i am checking if a Mean Deviation is relatively small to a full text length. If yes , then the text is distributed relatively evenly.

There is the code! https://github.com/Gelembjuk/articletext

Usage:

package main

import (
    "fmt"
    "os"
    "github.com/gelembjuk/articletext"
)

func main() {

    url := os.Args[1]
    text, err := articletext.GetArticleTextFromUrl(url)

    fmt.Println(text)
}

Add new comment

Last Updated on Wednesday, 06 July 2016 17:11

Implementing web site sign-up using popular social networks login in PHP

Majority of web sites use sign-up with social networks as alternate for traditional registration/login. Buttons “Login with: Facebook, Twitter, Google,... ” are present together with a login form very often now.

Using social networks for login simplifies a user experience and saves his time.

The author used social login on different web sites as well. Based on experience it was decided to build a PHP class/package that summarise the social login process on an abstract level and allows to implement the process with minimum coding.

Social login in common

For a user a social login looks like identical process in different social networks. Usually, it looks like: a user licks on a “login with ...” button. Is redirected to a social network website (http://facebook.com etc.) , clicks “Allow” on that site and, finally, is redirected back to an original website where his social login is processes and a session for a user is created.

However, on a back-end there is some difference.

Some social networks APIs don't need to connect to an api endpoint before to redirect a user a login page. Other has to connect to an API to init a login process, to get some special redirect url, to check some settings etc.

Next difference is that some APIs will require saving a specific data (request token etc) between 2 steps of an auth process – get a redirect url and complete login. For others no need to save something.

All such differences must be taken into account during a social login process.

Introduction gelembjuk/auth PHP package to unify the social login process.

The package is available in Composer. It allows to login with Facebook, Twitter, Google, LinkedIn with unified process and minimum coding. Additionally, the package can be easy extended with new social networks and same process will be used.

Installation

It is just normal installation with Composer

composer require gelembjuk/auth

Logger

gelembjuk/logger in composer, https://github.com/Gelembjuk/logger in Github.

The package contains classes for logging, error catching and display.

ErrorScreen class helps to catch different types of errors, PHP Exceptions, Warnings and Fatal errors, log error information and display correct error screen to a user depending on expected formats (html, json, xml, http).

FileLogger class based on Psr/Log helps to filter what to log. You can set On/Off different log groups. The main idea of this class is to help to debug big applications. But it would also work as a normal "everyday" logger to a file. It is used in my other packages/classes but can be replaced with any other Psr/Log logger.

ApplicationLogger trait helps to add logging functionality in different classes very easy. You don't need to care about such methods like log(),debug(), setLogger(), getLogger() etc. In your classes when this trait is used. Is really useful big big applications with many classes .

Will higher education system differ in future?

To get the answer on this question we need to understand if is everything all right in higher education. If modern higher education system is correct and everyone (or almost everyone) is happy with it then no need in changes. But if there are problems then we can expect changes.

Does modern higher education system have problems? If we care only about USA and West Europe (and other developed countries) then we could say “Higher Education will slowly transform under the influence of technology. All core principles will stay unchanged”. The reason is obvious. Developed countries built good higher education system. It filters gifted and talented children and allow them to get higher education for free or for low price. Such system seems even and, in general, satisfies the requirements of society. Of course, there are problems. On my opinion, on of biggest problems is not reasonable value for money . Nevertheless, imbalance is not sufficiently serious to accelerate changes.

What about developing countries? Population of such countries is bigger then in developed countries and there is no fair higher education system. Many gifted and talented children have no chance to get higher education. The have choice – pay big price to get good higher education or pay less money to get low quality education (formal higher education without real background) or get nothing and lose chance to go up with social ladder. What if they get some alternative that are cheaper then traditional higher education? They definitely need the alternative because something is better then nothing.

Latest News

Joomal Finance Tools

Download | Read more